{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# 使用Scikit-learn进行分类器之旅\n",
    "\n",
    "本章我们将要学习学术界和工业界常用的几种机器学习算法。在学习算法之间差异的同时，我们也要了解每个算法的优缺点。我们不会像上一章亲自实现各个算法，而是直接调用易用的scikit-learn库。\n",
    "\n",
    "本章涉及到的知识点包括：\n",
    "* 介绍常用分类算法的概念\n",
    "* 学习使用scikit-learn\n",
    "* 如何选择机器学习算法\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## 如何选择合适的分类器算法\n",
    "\n",
    "对于一个具体的分类问题，如何选择合适的分类器算法绝非纸上谈兵就能确定的：每一个算法都有其独特的数据偏好和假设。再一次强调\"No Free Lunch\"定理：在所有场景下都最厉害的分类器是不存在滴。实际上，常用的做法就是多选择几个分类器试试，然后挑选效果最好的那一个。对于同一个问题，样本数的不同、特征数目的多少、数据集中的噪音和数据是否线性可分 都会影响到分类器的效果。\n",
    "\n",
    "\n",
    "最终，分类器的性能、计算能力和预测能力，都极大依赖训练集。我们概况一下训练一个机器学习算法通常的5大步骤：\n",
    "* 特征选择\n",
    "* 选择性能评价指标\n",
    "* 选择分类器和优化算法\n",
    "* 评估模型的性能\n",
    "* 模型调参\n",
    "\n",
    "本书的内容贯穿上面的5大步骤，不要心急，本章重点关注常用算法的重要概念，至于特征选择、预处理、评价指标和如何调参将在后面章节一一介绍。\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## scikit-learn之旅\n",
    "\n",
    "在第二章，你学习了两个分类相关的算法：感知机和Adaline，并且都用Python进行了实现。现在我们学习scikit-learn的API，这个库不但用户界面友好并且对常用算法的实现进行了高度优化。此外，它还包含数据预处理和调参和模型评估的很多方法，是Python进行数据挖掘的必备工具。\n",
    "\n",
    "\n",
    "\n",
    "### 通过scikit-learn训练感知机模型\n",
    "\n",
    "我们先看一下如何使用sklearn训练一个感知机模型，数据集还是用我们熟悉的Iris数据集。由于sklearn已经内置了Iris数据集，所以本章所有的分类算法我们通通使用Iris数据集，还是和第二章一样，为了可视化方便，我们只使用其中两维度特征，样本数则使用三个类别全部的150个样本。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import numpy as np\n",
    "from sklearn import datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "iris = datasets.load_iris()\n",
    "X = iris.data[:, [2,3]]\n",
    "y = iris.target"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 1, 2])"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.unique(y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "为了评估训练好的模型对新数据的预测能力，我们先把Iris训练集分割为两部分：训练集和测试集。在第5章我们还会讨论模型评估的更多细节。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/lj/.pyenv/versions/2.7.12/lib/python2.7/site-packages/sklearn/cross_validation.py:44: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. Also note that the interface of the new CV iterators are different from that of this module. This module will be removed in 0.20.\n",
      "  \"This module will be removed in 0.20.\", DeprecationWarning)\n"
     ]
    }
   ],
   "source": [
    "from sklearn.cross_validation import train_test_split\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "通过调用train_tset_split方法我们将数据集随机分为两部分，测试集占30%(45个样本)，训练集占70%(105个样本)。\n",
    "\n",
    "许多机器学习和优化算法都要求对特征进行缩放操作，回忆第二章中梯度下降的例子，当时我们自己实现了标准化代码，现在我们可以直接调用sklearn中的StandardScaler来对特征进行标准化："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "sc = StandardScaler()\n",
    "sc.fit(X_train)\n",
    "X_train_std = sc.transform(X_train)\n",
    "X_test_std = sc.transform(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "上面的代码中我们先从preprocessing模块中读取StandardScaler类，然后得到一个初始化的StandardScaler新对象sc，使用fit方法，StandardScaler对训练集中**每一维度特征**计算出u (样本平均值)和$\\sigma$ (标准差)，然后调用transform方法对数据集进行标准化。注意我们用相同的标准化参数对待训练集和测试集。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "对数据标准化以后，我们可以训练一个感知机模型。sklearn中大多数算法都支持多类别分类，默认使用One-vs.-Rest方式实现。所以我们可以直接训练三分类的感知机模型:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Perceptron(alpha=0.0001, class_weight=None, eta0=0.1, fit_intercept=True,\n",
       "      n_iter=40, n_jobs=1, penalty=None, random_state=0, shuffle=True,\n",
       "      verbose=0, warm_start=False)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.linear_model import Perceptron\n",
    "\n",
    "ppn = Perceptron(n_iter=40, eta0=0.1, random_state=0)\n",
    "ppn.fit(X_train_std, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "\n",
    "sklearn中训练模型的接口和我们在第二章实现的一样耶，从linear_model模型读取Perceptron类，然后初始化得到ppn，接着使用fit方法训练一个模型。这里的eta0就是第二章中的学习率eta，n_iter同样表示对训练集迭代的次数。我们设置random_state参数使得shuffle结果可再现。\n",
    "\n",
    "训练好感知机模型后，我们可以使用predict方法进行预测了："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Misclassified samples: 4\n"
     ]
    }
   ],
   "source": [
    "y_pred = ppn.predict(X_test_std)\n",
    "print(\"Misclassified samples: {}\".format((y_test != y_pred).sum()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "\n",
    "对于测试集中45个样本，有4个样本被错分类了，因此，错分类率是0.089。除了使用错分类率，我们也可以使用分类准确率(accuracy)评价模型，accuracy=1-miscassification error = 1-0.089=0.911。\n",
    "\n",
    "Sklearn中包含了许多评价指标，这些指标都位于metrics模块，比如，我们可以计算分类准确率:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.911111111111\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "print(\"Accuracy: {}\".format(accuracy_score(y_test, y_pred)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "\n",
    "**Notes** 本章我们评估模型性能的好坏仅仅依赖于其在测试集的表现。在第5章，你将会学习许多其他的技巧来评估模型，包括可视化分析来检测和预防过拟合(overfitting)。过拟合意味着模型对训练集中的模式捕捉的很好，但是其泛化能力却很差。\n",
    "\n",
    "\n",
    "最后，我们使用第二章的plot_decision_regions画出分界区域。不过在使用之间，我们进行一点小修改，我们用圆圈表示测试集样本：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "from matplotlib.colors import ListedColormap\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "def plot_decision_regions(X, y, classifier, test_idx=None, resolution=0.02):\n",
    "    # setup marker generator and color map\n",
    "    markers = ('s', 'x', 'o', '^', 'v')\n",
    "    colors = ('red', 'blue', 'lightgreen', 'gray', 'cyan')\n",
    "    cmap = ListedColormap(colors[:len(np.unique(y))])\n",
    "\n",
    "    # plot the decision surface\n",
    "    x1_min, x1_max = X[:, 0].min() - 1, X[:, 0].max() + 1\n",
    "    x2_min, x2_max = X[:, 1].min() - 1, X[:, 1].max() + 1\n",
    "    xx1, xx2 = np.meshgrid(np.arange(x1_min, x1_max, resolution),\n",
    "                            np.arange(x2_min, x2_max, resolution))\n",
    "    Z = classifier.predict(np.array([xx1.ravel(), xx2.ravel()]).T)\n",
    "    Z = Z.reshape(xx1.shape)\n",
    "    plt.contourf(xx1, xx2, Z, slpha=0.4, cmap=cmap)\n",
    "    plt.xlim(xx1.min(), xx1.max())\n",
    "    plt.ylim(xx2.min(), xx2.max())\n",
    "\n",
    "    # plot all samples\n",
    "    X_test, y_test = X[test_idx, :], y[test_idx]\n",
    "    for idx, cl in enumerate(np.unique(y)):\n",
    "        plt.scatter(x=X[y == cl, 0], y=X[y == cl, 1],\n",
    "                    alpha=0.8, c=cmap(idx),\n",
    "                    marker=markers[idx], label=cl)\n",
    "\n",
    "    # highlight test samples\n",
    "    if test_idx:\n",
    "        X_test, y_test = X[test_idx, :], y[test_idx]\n",
    "        plt.scatter(X_test[:, 0], X_test[:, 1], c='',\n",
    "                    alpha=1.0, linewidth=1, marker='o',\n",
    "                    s=55, label='test set')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "经过上面的修改，现在可以区分训练集和测试集，运行如下代码："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "X_combined_std = np.vstack((X_train_std, X_test_std))\n",
    "y_combined = np.hstack((y_train, y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAEKCAYAAAASByJ7AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XmUVOW19/HvpptRxgYcUUlCQkRfBcU4oMYhcR6W88WY\nXKI3xKVGOjjH+yZE711J9MrrGAkOITEGg2AuxmhEBYfghKAoikSjMhgRBJnH7t7vH+dUU910Vx26\n6tSp4fdZq1bVGeo5u1qpXc94zN0RERFpl3QAIiJSHJQQREQEUEIQEZGQEoKIiABKCCIiElJCEBER\nQAlBRERCSggiIgIoIYiISKg66QB2RB8z7590EALAssF7Jh2CiES0+M3Fn7t732znlVRC6A+8nnQQ\nEnhzMbevvC3pKEQkgtqa2oVRzlOTkbTZqJpaamtGJR2GiOSJEoLkxLGkQxCRPFFCkJyNqqlNOgQR\nyYOS6kNoydZevVgyZgybBgyAdspvTTQ00OmDD+g3Zgztv/gi1kvV1ozitpW3x3oNEYlXySeEJWPG\n0O0b36B/dbUaL5pxYEVNDUvGjOFLo+Jt63cMalBHs0gJK/mf1JsGDKC3kkGLDOhdXR3UngpEncwi\npavkEwLt2ikZZGBQ0KY0x5QUREpU6ScEKToaeSRSmpQQ8uRvL73EwLPPZsCZZ/LLCROSDidxGnkk\nUnqUEPKgvr6ey26+mSdvv513J01i4rRpvPvhh0mHlTglBZHSUvKjjHbI974HLQ2/7NULfv/7Nhf7\n2jvvMGDPPflyv34A/Nu3v83U559n0Je/3OYyy8WomlqNPBIpEZWVEL74Ivjyb2l/Dj5Zvpw9d9ml\ncbvfLrvw6rx5OZVZTkbV1GK45imIFDk1GUlBqKNZpPgpIeTBHn37svizzxq3l3z2GXv0zbrSbMVR\nn4JIcVNCyIODBw3i/UWL+OiTT9iydSsPP/00px91VNJhFSXNURApXpXVhxCT6upq7rrmGk644grq\n6+u56PTT2fcrX0k6rKKkJS5EildlJYRevVofZZSjk4cN4+Rhw3Iup1JoMTyR4lNZCSGHoaWSX45h\nNRp5JFJM1IcgidHII5HiooQgidLII5HioYQgiVNSECkOSghSFJQURJKXWEIwsz3NbIaZvWtm75iZ\nBqhXOM1REElWkjWEOuBKdx8EHApcZmaDEoynzS668UZ2Pv549jv//KRDKWmOqaYgkqDEEoK7f+ru\nc8LXa4H5wB7xXzfzdluMOPVU/nbHHbkXJICaj0SSUhR9CGbWHxgCvNrCsZFm9rqZvb48x+uMf7QP\nYx/apTEJuMPYh3Zh/KN9cir3qAMPpKZ79xyjk3RqPhIpvMQTgpl1BaYAte6+pvlxdx/v7kPdfWgu\ny8W5w9oNVUycVtOYFMY+tAsTp9WwdkNVXmoKkj9qPhIpvERnKptZe4Jk8JC7PxrvtWD0d4IVSSdO\nq2HitBoAhh+/ktHf+QzTHKmipCUuRAonyVFGBtwPzHf3sYW55rakkKJkUNwcU/ORSIEk2WQ0DPgu\ncKyZvRk+To7zgqlmonTpfQpSnLTEhUhhJDnK6O/ubu6+v7sPDh9PxHe9bX0Gw49fyazfzWf48Sub\n9Cm01fAbbuCwiy5iwcKF9DvlFO6fOjV/gQugkUcihVAxq52aQbcu9U36DFLNR9261OfUbDTxv/87\nT1FKJro3s0i8KiYhAIw863PcafzyTyUF9SGUDse4Hd1cRyQOiQ87LbTmX/5KBqVHzUci8ai4hCDl\nQSOPRPJPCUFKkiauieSfEoKIiABKCFLiRtXUqqYgkidKCHmweOlSjrnkEgaddx77nncet0+cmHRI\nFUdJQSR3Sgh5UF1dza21tbw7aRKv/Pa33D15Mu9++GHSYVUcdTSL5KbiEsLHnVcyZbd5/Hav15my\n2zw+7rwy5zJ369OHA7/+dQC67bQT+/TvzyfLc12sW3aUlrgQyU1FJYSPO69kRt8PWV+1mY71Vayv\n2syMvh/mJSk0XuNf/+KNBQs4ZN9981amRKemI5G2a3WmspmdFeH9m+JcfyjfZvf8F1UNRnuvAgie\nG+qZ3fNf9N9Yk3P56zZs4Oxrr+W20aPp3rVrzuVJ22iJC5G2ybR0xb3AVMhYDz8KKJmEsKb9JjrW\nVzXZV+3tWNN+U85lb62r4+xrr+U7J57IWccem3N5khstcSGy4zIlhCfd/aJMbzazP+Q5nlh139qJ\n9VWbG2sIAHXWQPetnXIq1925+Kab2Kd/f0Z/5zu5hil5MqqmlttXKimIRNVqH4K7X5jtzVHOKSYH\nrdqd+nbOVqvHCZ7r2zkHrdo9p3Jnzp3Lg088wfTXX2fwBRcw+IILeGLmzDxFLbnQyCOR6NrchxD3\nLS/j0H9jDccsD/oS1rTfRPetnThoxe459x8cMXgwPmtWnqKUfHIMalBNQSSCTE1Gp4XPOwOHA9PD\n7WOAl4CSSwgQJIV8dCBLadG9mUWyazUhuPv3AcxsGjDI3T8Nt3cDJhQkOpE8UU1BJLso8xD2TCWD\n0GfAXjHFIyIiCYlyx7RnzewpILVAz/nAM/GFJBIfzVEQaV3WGoK7Xw6MAw4IH+Pd/UdxByYSFy1x\nIdKyqEtXzAH+6u4/Bp4ys24xxiQSOy1xIbK9rAnBzH4ATAZ+E+7aA/jfOIMqJavWruXXjzzS5vff\n9sc/smFT7jOln5s9m5fmzs25nEoyqqZW8xRE0kSpIVwGDAPWALj7+wRDUYUwIUye3Ob33/bww/lL\nCG+9lXM5lUbNRyLbROlU3uzuW8yCfzhmVg14rFGVkOvuuot/fvIJgy+4gG8fcgi3jBrFLQ8+yKSn\nn2bz1q2cefTR/PyHP2T9xo2cd/31LFm2jPr6ev7vxRfz2cqV/Gv5co655BL69OzJjHHjmpZ95508\n9uKLVFdVcfwhh/A/tbUs/+ILLvnFL1i0dCkAt115JXv07cu4KVOoqqriD08+yZ1XX82RQ4Yk8eco\nSVriQiQQJSE8b2Y/ATqb2beBS4G/xBtW6fjl5Zcz75//5M0//hGAaa+8wvuLFvHa736Hu3P6lVfy\nwpw5LF+1it379OGvtwVfPKvXraNH166M/eMfmTFuHH169mxS7opVq/jzc8/x3uTJmBmr1q4FYNSt\nt/LjCy7giMGDWbR0KSf86EfMf+QRLjn7bLp27sxV3/1uYf8AZUIT10SiJYTrgIuBt4EfAk+4+72x\nRlXCpr3yCtNefZUh4SJ36zZu5P3Fizly8GCuvO02rr3zTk494oisv+B7dO1Kp44dufimmzj1iCM4\n9cgjAXjmtdea3I1tzfr1rNuwIb4PVCE0cU0kWkL4DvBwehIws1Pd/fH4wipd7s71I0bww7O2Xwpq\nzoMP8sTMmfznPfdw3MEH89Mf/KDVcqqrq3ltwgSenTWLyc8+y12PPML0e+6hoaGBV377Wzp17Bjn\nx6hYqilIJYvSqXwn8KKZ7ZO278aY4ik53bp0YW3aL/QTDjuMBx57rPFX+yfLlrEs7Cvo0qkTF558\nMld/97vMWbBg2/vXr9+u3HUbNrB63TpOHjaM/zd6NHPffx+A4w89lDsnTWo87830clRTyJljGpIq\nFStKDeEjgiajyWY2xt0fIfNNcypK7549GXbAAex3/vmcdPjh3DJqFPM/+ojDLgpuJdG1Sxf+cOON\nfLB4MVffcQftzGhfXc09110HwMgzz+TEK65g9759m3Qqr92wgTOuvJJNW7bg7oytDb6k7rjqKi77\n1a/Yf/hw6urrOWrIEMZdfz2nHXkk51x3HVOff16dyiLSJuaeecCQmc1x9wPNrA/B8hVzgePdff9C\nBJhuqJm/3mzf/CefZJ8+fQodSkmZ//nn7HPSSUmHUXK0xIWUi9qa2tnuPjTbeVGajD4FcPfPgRMI\nhpzul1t4IsXPMU1ck4oSZS2jU9JeN7j71e4edckLkZKmiWtSSVr9Yjez28Lnv5jZY80fhQsxi4YG\nzZLLwAEaGpIOo6SpliCVIlOn8oPh8//EdXEzewA4FVjm7m1qhur0wQesqKmhd3W1fss148CKujo6\nffBB0qGUNM1RkEqRtVM51oubHQWsA34fJSG01Km8tVcvlowZw6YBA6CdWrKaaGig0wcf0G/MGNp/\n8UXS0ZQFJQUpRVE7lVtNCGb2NhnWLMrXKCMz6w883taEIFJIGnkkpShqQsjUZHRq+HxZ+JxqQrqQ\nAi5uZ2YjgZGg+3ZK8tR8JOWs1TYWd1/o7guBb7v7Ne7+dvi4Fji+UAG6+3h3H+ruQ/sW6qIiWWg2\ns5SjKI3uZmbD0jYOj/g+kbKm0UdSbqJ8sV8E/NrMPjazj4Ffh/tEKprmKEi5ybiWkZm1Awa4+wFm\n1gPA3Vfn6+JmNhE4GuhjZkuAn7n7/fkqXyRuo2pq1dEsZSNjQnD3BjO7BpiUz0SQVv7wfJcpUmiO\nYTVKClL6ojQZPWNmV5nZnmZWk3rEHplICVHzkZSDKAnhfIKhpy8As8OHpgOINFPokUfNpxAlOMdU\nykSUxe2+1MLjy4UITqTUFCopvDRhPz56ZZfGJOAODXWmZaskJ5GGj5rZfmZ2npl9L/WIOzCRUhV3\nUnCHXQcuZ++Dl9FQv61m0K7awVVTkLbLmhDM7GcEt9G8EzgGuBk4Pea4REpanHMUzOBLh34GOA11\nVU3WDbB2wXGRtohSQzgHOA5Y6u7fBw4AesQalUiJi/vezGbQriq10XS/SFtFSQgb3b0BqDOz7sAy\nYM94wxIpD3ElBXdoqE9tNN0v0lYZ5yGEXjeznsC9BCOM1gEvxxqVSJkwHMJB2vmap+AOH72yC3sf\nvIx21fVNagjeQOPPvPTagrtqD5JdlFFGl7r7KncfB3wb+Pew6UhEdkC++hXMYOmCviyctTPtqrZ9\n0TfUGRjUbWm33QikVUu6smlt+7xcX8pXpvshHJjpje4+J5aIMtD9EKTUWJaV4nOpNTT/1Z/6p/zc\n3UOY8+hADjxrAUdf9sZ226opVJ583A/h1vC5EzAUmEtQOd2fYGLaYbkGKVLp0msNO5ocmn+xp7aP\nvuwNAOY8OpA5jw4EUDKQSDLdD+EYdz8G+BQ4MLwnwUHAEOCTQgUoUqqy1Q6aW7Gwe36ua9uSQoqS\ngUQRZZTRQHd/O7Xh7vOAfeILSaQyrVjYnaXv9WLRGzuz6I2dgaajhqIuVeEeNBs1307vUxBpSZSE\n8JaZ3WdmR4ePe4G34g5MpNJMv+MgqjvW06HzVsBZ+l4vVnzcnXHnnMFLE/bb7kv9ubuH8NKEprci\nT+1P9Rkc+r159Nh1HbOnDOS5u4fQ0NDy+0QgWkL4PvAOMCp8vBvuE5FW7GhzEcB70/sz+epjWLNs\nJ3aq2cyaZTsx+epjeG96fyaNPo6pPz2KH/ce1eRLf/O69k1+8ZtBx65bOfCsBXzz0jfYsq49q5d2\npceu6+iw01ae/3XL7xOBDKOMipFGGUkpaEsyaIszbnyh1b6B1AikVPKYPWVg43nqYK48UUcZRVnL\naJiZPW1m/zCzD1OP/IQpIm2VqjG0NL8h9WWf6mBO//JXMpDWRJmpfD/wY4JZyvVZzhWRBLQ2fLV5\nBzME20oK0pIofQir3f1Jd1/m7itSj9gjEylBhWouyqS2ZhQNDdt3MI9+9mEOPGsBcx4d2KSDWiQl\nSg1hhpndAjwKbE7tTGKmsohEM/68Mzj2itnsf9o/GXr+u3Tts7mx+ahmr9WsX9lFNQTZTpSEcEj4\nnN4h4cCx+Q9HRHL19WM/5pxbZlC3uYr6rdC5x1bqtxorF3dj56+sYf/TPlQykBZlTQjhbGURyaIY\nmosAjr1iNnWbq9iysT19B6yjocEwN7rvvJFFb+xMh85bWbNsJ7521JKkQ5UiE6WGgJmdAuxLsK4R\nAO5+Y1xBiUjb9d57DetXdgSgfYd66usNx6juEIwJ2bKxmt57r2nsiM7XstxS+rImBDMbB3QhuH3m\nfQR3UHst5rhEip7T5FYERWPFwu5033k9Wza2Z+uWKqraN2Du1G0JbrHWoXNdk3WTcllgT8pLlFFG\nh7v794Av3P3nBKucfi3esESK23h+wFhGNzYSFUdjUSB9CYy1n3WmXTvHqhpYs6wzHTpvpbpjPdPv\nOKjF99bWjGp8SOWJdAvN8HmDme0ObAV2iy8kkeLmwFq6MZHhjUmhXRGlhPQlMKraw9IFvfhsQW+q\nqmmyHEY2SgqVJ0ofwuPhLTRvAeYQ/Hu4L9aoRIqYAaMZC8BEhjOR4ckG1IL3pveP9KWfTSopXPq/\nU9QJXQGiJISb3X0zMMXMHifoWN4Ub1gixS2VFIoxGcSh995rWPRGX9J7TfYasiy5gCQWUZqMXk69\ncPfN7r46fZ9IJXJgLKMBmE3WNcNK3oqF3enQua7JvqXv9UooGolLqzUEM9sV2APobGZD2PbToDvB\nqCORstN85FBLI4lSyWAiwysiGUDQUX3OLTOAYNhqh851VHesZ9w5ZzQ2TWmEUunL1GR0AjAC6Edw\nf+XUv4u1wE/iDUuk8MbzA9bSjdGMxdj2xd+NtYzk3sbzDOjGWoYzsWISQqqj+tgrZtN77zWsWNid\n6Xcc1KSfQsNXS1/W+yGY2dnuPqVA8WSk+yFIXNJ/9Q9nIqMZu912SzWFYhpdVKyUHJKXt/shAP3M\nrLsF7jOzOWZ2fB5iFCkaqU7i4UxkIsM5mFkZkwEoGUSl4aulI0oNYa67H2BmJwCXAP8JPOjuB+Z8\ncbMTgduBKuA+d/9lpvNVQ5C4OXAwsxq3Z3Fwq7ORi2XtolKkWkNh5bOGkPr3cDLwe3d/hzzM2Dez\nKuBu4CRgEDDczAblWq5IW6WPHEpJn40s+aMZ0cUpSkKYbWbTCBLCU2bWDWjIw7W/AXzg7h+6+xbg\nYeCMPJQrssOa9yHM4uDG5qOWksJ4fpBEmGVJSaF4RJmYdjEwGPjQ3TeYWW/g+3m49h7A4rTtJWy7\n94JIQaWPHEr1GaRmI3dj7XZDUX/I+ASiLF8aoVQcMs5DcPel7t5AsGQFAOHtM1eknxNngGY2EhgJ\nsFecF5KKN5J7m8w7SCWF5u2jxbjCaTlRckhOphrCE0C2juMo57TmE2DPtO1+4b4m3H08BD/Hhpqp\nOVdipS//4qLkUFiZEsIBZrYmw3EDMh3PZhbwVTP7EkEi+DfgghzKEykIjS5Khm7oE79WE4K7V8V5\nYXevM7PLgacIhp0+EI5gEokkyjIT+b6e5h4kT7WG+ES6hWZc3P0JgmYnkR0SdZmJfF9PiouSQ35F\nGXYqUlRaukFNasjoWrrl/Td8+vWkeGn4au6yzlQuJpqpLCnpSSAl0zIT+biemotKi2oM20SdqRwp\nIYSzinchrYnJ3RflFGEbKCFIuh1ZZiIf1Jlc2io5QeRt6Qoz+xHwGfA08Nfw8XjOEYrkYEeXmWg+\ntb4h4jFv9iylS0tlZBelU3kUMDCckCaSuExLVcP2k8ku5PespgdTOYN2BF/4ZzCVHqwGaPXYUbzI\nWrrxY8ZSpZRQNjR8tXVREsJiCP/liBSBHVlmooHgC386x3EGU5nKGZzBVKZzHMfwLAAzWjm2hm48\nrI7ksqURSttrtQ/BzFL18X2BgQRNRZtTx919bOzRNaM+BEkXdR5C6lf/dI5r3HcszzI1XEuxtWNG\n5d0qU8ozOeTcqWxmP8vwPnf3G9saXFspIUhbNQDdWNe4vZaujR1omY6lOq6VECpLuSWFqAkh00zl\nnwOY2bnu/kj6MTM7N/cQRQojVUNIl2oiIsOxVA1ByaDyNO98LrcE0ZoofQjXA49E2CdSdNKbi1JN\nQant08NEMKOVY0fzvPoQBKic/oZMy1+fRHBTnD3M7I60Q92BurgDE8mHdkAPVjd+4beDxi/+1Cij\n1o51DzuuVUOQdOU8SilTH8IBwBDg58BP0w6tBWa4+xfxh9eU+hCkrRpoOukmfTvTMU1Gk2xKITHk\now9hLjDXzB5y9615jU6kwJrPwGwX4ZiSgURRTs1JmZqM3iacoGm2/WA+d98/vrBEREpPqSeHTJ3K\np4bPl4XPD4bPF6KZ/CIiGZVicsi6uJ2ZveHuQ5rtm+Pubb11ZpupD0EKRc1FEoekEkPOfQhpzMyG\nufvMcONwdB8FEZEdVuy1higJ4WLgATPrQbAywBfARbFGJSJS5opx+GrWhODus4EDwoSAu2uhOylr\nai6SQiqmxJBplNGF7v6HtEXuUvuBZBa3ExEpV8XQnJSphrBT+Kw7i4uIFFBSySHTxLTfhC9/5e6b\nChSPiIikKWRyiNKpPM/MPgNeDB9/Vz+ClCv1H0gxi7u/IUqn8gAz2ws4EjgFuNvMVrn74FgiEhGR\njOKqNWRNCGbWDxhGkBAOAN4B/p63CEREpM3ymRyiTDBbBNQCT7r7Ye5+irv/IqerihQhNRdJqWt+\nY58dFWXpigOAI4CjgL2A94Hn3f3+nK7cBlq6QuKiZCDlzfKzdIW7zzWzfwL/JGg2uhD4JlDwhCAi\nIvGJ0ofwOtAReIlglNFR7r4w7sBERKSwogw7Pcndl8ceiUhC1FwkEsjaqaxkICJSGbSMtYiIAEoI\nIiISyrTa6VmZ3ujuj+Y/HJHCUv+ByDaZOpVPy3DMgTYnBDM7FxgD7AN8w13TC0REkpZptdPvx3jd\necBZwG+ynSgiIoURZdgpZnYKsC/QKbXP3W9s60XdfX5YbluLEMmZmotEmsraqWxm44DzgR8R3FP5\nXGDvmOMSEZECi1JDONzd9zezt9z952Z2K/BktjeZ2TPAri0cusHdp0YN0MxGAiMhWEhJRETiESUh\nbAyfN5jZ7sAKYLdsb3L3b+USWFo544HxECxul48yRdRcJLK9KAnhcTPrCdwCzCEYYXRfrFGJxEjJ\nQKRlUZa/7ujum1OvCTqWN6X2temiZmcCdwJ9gVXAm+5+Qrb3aflryQclBKk80Za/jjJT+eXUC3ff\nHN5P+eUM52fl7n92937u3tHdd4mSDEREJF6ZZirvCuwBdDazIQQjjAC6A10KEJtI3ql2INK6TH0I\nJwAjgH7A2LT9a4CfxBiTiIgkINNM5d8BvzOzs919SgFjEhGRBETpQ5hpZveb2ZMAZjbIzC6OOS4R\nESmwKAnht8BTwO7h9j+A2tgiEomJ+g9EMouSEPq4+ySgAcDd64D6WKMSEZGCi5IQ1ptZb4IJaZjZ\nocDqWKMSEZGCizJTeTTwGPAVM5tJMJnsnFijEskzNReJZJc1Ibj7HDP7JjCQYC7CAnffGntkInmi\nZCASTdaEYGadgEuBIwiajV40s3Huvinu4EREpHCiNBn9HlhLsPYQwAXAgwT3RRARkTIRJSHs5+6D\n0rZnmNm7cQUkkk9qLhKJLsoooznhyCIAzOwQQIuOioiUmSg1hIOAl8xsUbi9F7DAzN4G3N33jy06\nEREpmCgJ4cTYoxARkcRFGXa6sBCBiOSb+g9EdkyUPgQREakASggiIgIoIUiZUnORyI5TQhAREUAJ\nQUREQkoIUnbUXCTSNkoIUlaUDETaTglBREQAJQQREQkpIUjZUHORSG6UEEREBFBCEBGRkBKCiIgA\nSghSJtR/IJI7JQQREQGUEEREJKSEICVPzUUi+aGEICIiQEIJwcxuMbP3zOwtM/uzmfVMIg4REdkm\nqRrC08B+7r4/8A/g+oTikBKn5iKR/EkkIbj7NHevCzdfAfolEYeUNiUDkfwqhj6Ei4Ankw5CRKTS\nVcdVsJk9A+zawqEb3H1qeM4NQB3wUIZyRgIjAfaKIU4REQmYezLVbjMbAfwQOM7dN0R5z1Azfz3W\nqKSUqMlIJCqb7e5Ds50VWw0hEzM7EbgG+GbUZCCSTslAJP+S6kO4C+gGPG1mb5rZuITiEBGRUCI1\nBHcfkMR1RUSkdcUwykhkh6i5SCQeSggiIgIoIYiISEgJQUqKmotE4qOEICVDyUAkXkoIIiICKCGI\niEhICUFKgpqLROKnhCAiIoASgoiIhJQQpOipuUikMJQQREQEUEIQEZGQEoKIiAAJ3jGtLcxsObAw\n6TjS9AE+TzqIBFXy56/kzw6V/flL8bPv7e59s51UUgmh2JjZ61FuS1euKvnzV/Jnh8r+/OX82dVk\nJCIigBKCiIiElBByMz7pABJWyZ+/kj87VPbnL9vPrj4EEREBVEMQEZGQEkKOzOwWM3vPzN4ysz+b\nWc+kYyokMzvXzN4xswYzK8uRF82Z2YlmtsDMPjCz65KOp5DM7AEzW2Zm85KOpdDMbE8zm2Fm74b/\nz49KOqZ8U0LI3dPAfu6+P/AP4PqE4ym0ecBZwAtJB1IIZlYF3A2cBAwChpvZoGSjKqgJwIlJB5GQ\nOuBKdx8EHApcVm7/7ZUQcuTu09y9Ltx8BeiXZDyF5u7z3X1B0nEU0DeAD9z9Q3ffAjwMnJFwTAXj\n7i8AK5OOIwnu/qm7zwlfrwXmA3skG1V+KSHk10XAk0kHIbHaA1ictr2EMvtSkOzMrD8wBHg12Ujy\nqzrpAEqBmT0D7NrCoRvcfWp4zg0EVcqHChlbIUT5/CKVwsy6AlOAWndfk3Q8+aSEEIG7fyvTcTMb\nAZwKHOdlOI432+evMJ8Ae6Zt9wv3SQUws/YEyeAhd3806XjyTU1GOTKzE4FrgNPdfUPS8UjsZgFf\nNbMvmVkH4N+AxxKOSQrAzAy4H5jv7mOTjicOSgi5uwvoBjxtZm+a2bikAyokMzvTzJYAhwF/NbOn\nko4pTuEAgsuBpwg6FSe5+zvJRlU4ZjYReBkYaGZLzOzipGMqoGHAd4Fjw3/rb5rZyUkHlU+aqSwi\nIoBqCCIiElJCEBERQAlBRERCSggiIgIoIYiISEgJQQrGzEaY2e4RzptgZudE3Z+HuH6S9rp/lJU8\nw1g+MrNLMpwzOJ/DEsO/3105lvFcalVaM3si19V5zexoM3s8fH1+uALs47mUKclRQpBCGgFkTQgJ\n+En2U1p0tbtnmncyGEhsnLqZZVyJwN1PdvdV+bqeu/8J+I98lSeFp4QgbRL+kn7PzB4ys/lmNtnM\nuoTHDjKz581stpk9ZWa7hb/shwIPhRN6OpvZT81slpnNM7Px4UzQqNff7hrh/ufM7Fdm9pqZ/cPM\njgz3dzHCvD8HAAAEa0lEQVSzSeFa9n82s1fNbKiZ/RLoHMaUWoeqyszuDde8n2ZmnSPEc274Oeaa\n2QvhLOYbgfPDss83s2+Y2ctm9oaZvWRmA8P3jjCzR83sb2b2vpndnFbu98PP8RrBxKjU/tPCz/CG\nmT1jZruE+8eY2YNmNhN4MPw7Pxz+N/oz0DmtjI/NrI+ZXZI20eojM5sRHj8+jHeOmT1iwRo+qftB\nvGdmcwiWPpdy4e566LHDD6A/4MCwcPsB4CqgPfAS0Dfcfz7wQPj6OWBoWhk1aa8fBE4LX08Azmnh\nmhOAcyJc49bw9cnAM+Hrq4DfhK/3I1iIcGi4va7Z56oDBofbk4ALW4slbfttYI/wdc/weQRwV9o5\n3YHq8PW3gClp530I9AA6AQsJ1kvaDVgE9AU6ADNT5QG92Dax9D/SPvMYYDbQOdwenfa32b/Z5/4Y\n6JMWX3vgReA0oA/BPS52Co9dC/w0jG8x8FXAwr/P42llHJ2+rUdpPbS4neRisbvPDF//AbgC+BvB\nF+7T4Q/+KuDTVt5/jJldA3QBaoB3gL9EuO7ALNdILTo2m+ALHuAI4HYAd59nZm9lKP8jd3+zhTIy\nmQlMMLNJaddvrgfwOzP7KkEybZ927Fl3Xw1gZu8CexN8KT/n7svD/X8Cvhae3w/4U1gz6gB8lFbW\nY+6+MXx9FHAHgLu/leVz3w5Md/e/mNmpBDcAmhn+jTsQLFnxdYK/z/thTH8ARmYoU0qIEoLkovm6\nJ07wq/Eddz8s0xvNrBPwa4Jfq4vNbAzBr88osl1jc/hcT9v+H9+c9rqetGaW1rj7JWZ2CHAKMNvM\nDmrhtJuAGe5+pgXr6T+X4ZrZ4r4TGOvuj5nZ0QQ1g5T12eJtzoIVe/cmWKcJgr/x0+4+vNl5g3e0\nbCkd6kOQXOxlZqkv5QuAvwMLgL6p/WbW3sz2Dc9ZS7AQIGz78v88bJvekdFDma7RmpnAeeH5g4D/\nk3ZsqwXLGreZmX3F3V91958CywmafNI/LwQ1hNRS2SMiFPsq8E0z6x3Gd24rZf17hjJeIPhvg5nt\nR9Bs1Dz2gwia1C5094Zw9yvAMDMbEJ6zk5l9DXgP6G9mXwnPG968PCldSgiSiwUE95WdT9CmfY8H\nt5U8B/iVmc0F3gQOD8+fAIwzszcJfhHfS3BP5qcIlpWOJMs1WvNrgiTyLvBfBM1Tq8Nj44G30jqV\n2+IWM3vbgiGrLwFzgRnAoFSnMnAz8Asze4MINRd3/5Tgl//LBAltftrhMcAjZjYb+DxDMfcAXcP/\nRjcSNIE1dzlBk92MMNb7wmaqEcDEsJnpZeDr7r6JoInor2Gn8rJsn0NKh1Y7lTYJmzwed/f9Eg4l\nEjOrAtq7+6bw1+0zwMAwubSlvAkEn39yHsMseWHz1VXufmrSsciOUx+CVIouBL+A2xO0j1/a1mQQ\nWg3cZGZ9PPNchIoR1oJ+Rsu1ECkBqiGIiAigPgQREQkpIYiICKCEICIiISUEEREBlBBERCSkhCAi\nIgD8f1s8Ar87WjkSAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x113160ad0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_decision_regions(X=X_combined_std,\n",
    "                     y=y_combined,\n",
    "                     classifier=ppn,\n",
    "                     test_idx=range(105, 150))\n",
    "plt.xlabel('petal length [standardized]')\n",
    "plt.ylabel('petal width [standardized]')\n",
    "plt.legend(loc='upper left')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "\n",
    "对结果可视化后，我们可以发现三个类别的花不能被线性决策界完美分类。\n",
    "\n",
    "在第二章我们就说过，感知机对于不能够线性可分的数据，算法永远不会收敛，这也是为什么我们不推荐大家实际使用感知机的原因。在接下来的章节，我们将学习到其他线性分类器，即使数据不能完美线性分类，也能收敛到最小的损失值。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## 逻辑斯蒂回归对类别概率建模\n",
    "\n",
    "感知机算法为我们学习机器学习分类问题曾经立下汗马功劳，但由于其致命缺点：**如果数据不能完全线性分割，则算法永远不会收敛**。我们实际上很少真正使用感知机模型。\n",
    "\n",
    "接下来我们学习另一个非常有效的线性二分类模型：逻辑斯蒂回归(logistic regression)。注意，尽管模型名字中有“回归”的字眼，但她确是百分百的分类模型。\n",
    "\n",
    "### 逻辑斯蒂回归和条件概率\n",
    "\n",
    "逻辑斯蒂回归(以下简称逻辑回归)是一个分类模型，它易于实现，并且对于线性可分的类别数据性能良好。她是工业界最常用的分类模型之一。和感知机和Adaline相似，本章的逻辑回归模型也是用于二分类的线性模型，当然可以使用OvR技巧扩展为多分类模型。\n",
    "\n",
    "逻辑回归作为一个概率模型，为了解释其背后的原理，我们先介绍一个概念：几率(odds ratio)=$\\frac{p}{(1-p)}$, 其中p表示样本为正例的概率。这里如何划分正例、负例要根据我们想要预测什么，比如，我们要预测一个病人有某种疾病的概率，则病人有此疾病为正例。数学上，正例表示类别y=1。有了几率的概念，我们可以定义对数几率函数(logit function,这里log odds简称logit)：\n",
    "\n",
    "$$logit(p)=log\\frac{p}{(1-p)}$$\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "对数几率函数的自变量p取值范围[0,1],因变量值域为实数域，将上式中的p视为类后验概率估计p(y=1|$x$),然后定义如下线性关系：\n",
    "\n",
    "$$logit(p(y=1|x)) = w_{0}x_{0} + w_{1}x_{1}+...+w_{m}x_{m}=\\sum_{i=1}^{m}w_{i}x_{i}=\\mathrm{w}^{T}\\mathrm{x}$$\n",
    "\n",
    "\n",
    "实际上我们关心的是某个样本属于类别的概率，恰恰是对数几率函数的反函数，也被称为逻辑斯底函数(logistic function)，有时简写为sigmoid函数，函数图像是S型：\n",
    "\n",
    "$$\\phi(z)=\\frac{1}{1+e^{-z}}$$\n",
    "\n",
    "其中z是网络输入，即权重参数和特征的线性组合$z=w^{T}x=w_{0}+w_{1}x_{1}+...+w_{m}x_{m}$。\n",
    "\n",
    "\n",
    "sigmoid(S曲线)函数很重要，我们不妨画图看一看：\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def sigmoid(z):\n",
    "    return 1. / (1.+np.exp(-z))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "z = np.arange(-7, 7, 0.1)\n",
    "phi_z = sigmoid(z)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYkAAAEKCAYAAADn+anLAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAHqlJREFUeJzt3Xl8VPW9//HXZ2ayEwIh7FvYN0HBgIreGve1brW3+qt1\nv7Qq6lW7aPtr621va61eW+r6c+vPpS51aQVLXQHrrxYFlMUAYQlLIOyBhOyZme/vj4wIyIQJkJxZ\n3s/HYx6ZOXNm8k4ek7zne86c7zHnHCIiIgfi8zqAiIjEL5WEiIhEpZIQEZGoVBIiIhKVSkJERKJS\nSYiISFQqCRERiUolISIiUakkREQkqoDXAQ5XQUGBKyws9DqGyAGVlpYCMGLECI+TiOxrwYIF251z\n3Q+2XsKXRGFhIfPnz/c6hsgBFRcXAzBnzhxPc4jsz8zWxbKeNjeJiEhUKgkREYlKJSEiIlGpJERE\nJCqVhIiIRKWSEBGRqFQSIiISlUpCRESiUkmIiEhUKgkREYlKJSEiIlGpJEREJCqVhIiIRNVhJWFm\nT5vZVjP7PMr9ZmZ/MLNVZrbYzCZ0VDYRETmwjhxJ/F/g7FbuPwcYFrlMAR7tgEwiItKKDjufhHPu\nH2ZW2MoqFwLPOuccMNfMuphZb+fcpg4JKCISR5xzhB2Ewo6wa7mEwo5wGEKR2znpAbLS/e2aI55O\nOtQXKN/r9obIsq+UhJlNoWW0wYABAzoknIikhlDYUdMYbLk0BKlpbKamMUR9U4jGYIiG5hANzWEa\nmkPU73W95b4wjcEQTUFHMBwmGHI0h8IEw45gKExz6MvlTaGWr8FwZHkoTHPYEQ47Qs7h3MGz/uri\no/j2cQPb9fcRTyURM+fc48DjAEVFRTH8KkUkFTU0h9i2u5EdtU1U1jayo6aJytqWy47aJnZGvu5u\naN5TCrVNoTZ9j3S/j4w0H5lpfjLTfGQE/KT5faT5jYDPCPh9ZKb5CGQEIst8BPz25Tp+H2mR9QJ+\nw2+G32f4Il/9PsOMryz3+YyJhfnt9Jv7UjyVxEag/163+0WWiYgcUG1jkDXba1mzvZaKXfVsqmpg\n4676Pdcra5sO+Lh0v4/8nHTyc9Lp1imdvl2yyM0M0CkjQKfI19zMALmZaXTKCJCT4Y+UgJ+sNP8+\nheD3WQf/1B0rnkpiOjDVzF4CjgOqtD9CRAB21jZRUlHNqq27Wb2tlrLtNazeWsvm6oZ91svNDNAn\nL4veXTI5un8X+uRl0iM3s6UQOqXTLVIMnTICmCX3P/cjpcNKwsxeBIqBAjPbAPwcSANwzj0GzATO\nBVYBdcA1HZVNROJHbWOQheW7WLBuJ59vrKKkopqNu+r33J+bGWBI905MHtqNId07MaR7DoUFOZHR\nQJqHyZNTR3666fKD3O+AmzoojojEidrGIB+t3sHcsh3MW1tJSUU1obDDDAYV5DBhYFeuPGEgY/rk\nMaJXLgWd0jUK6EDxtLlJRFLEyi27mV26lTml25i3tpLmkCMj4OOY/l244eQhFBV2ZcLArnTWyMBz\nKgkR6RBl22p4c/EmZiyqYOXWGgCG9+zENScOonh4d44t7EpGoH0/8y9tp5IQkXZTVd/M659u4NUF\nGyipqAZgUmE+v7hwDKeP6kmfLlkeJ5SDUUmIyBG3ZEMVz89dx/RFFdQ3hxjbN4//fd4ozhvXm955\nKoZEopIQkSPCOces5Vt5aPYqPlu/i6w0PxeN78O3jxvIUX3zvI4nh0glISKHJRx2vLN0Cw/OWklJ\nRTX9umZx99dHc8mx/bTjOQmoJETkkH2wYhv3zFzG8s27GVSQw32XjuOi8X1J8+tUNclCJSEibbZm\ney3//eZS3l++lUEFOUy77BjOH9cn6aeoSEUqCRGJWU1jkAffX8nT/1xDRsDPj88dydWTB5Ee0Mgh\nWakkRCQmH63ezg9eWUxFVT3fPLYf3z9rBD1yM72OJe1MJSEirWpoDnHvW8v54z/XMrggh9dumMyE\nAV29jiUdRCUhIlEtLN/F7X9eSNm2Wq6eXMiPzh7Z7mdCk/iikhCRr3DO8fzcdfzXjKV0z83gT9cf\nx4lDC7yOJR5QSYjIPhqDIX721xJenl/OqSN78LtvHUNelo53SFUqCRHZY0t1A997fgGfrd/F1FOG\ncvsZw/HpY60pTSUhIkDLfEvXPTOPmsYgj3x7AueO7e11JIkDKgkRYW7ZDq5/Zj55WWm8fuNkRvbq\n7HUkiRMqCZEUN2v5Fm54/lP652fz/HXH0StPxz7Il1QSIilsxqIKbnt5IaN6d+aZayeRn5PudSSJ\nMyoJkRT18rz13Pn6EiYW5vPUVUXkasZWOQCVhEgKmr6ogjtfX8LJw7vz2BXHkpmmA+TkwDQrl0iK\nmVO6ldtfXsjEwnwVhByUSkIkhSxYV8n3nl/AiF65PHlVkQpCDkolIZIilm2q5po/zqN3XhbPXDtJ\nZ42TmKgkRFLAxl31XPn0J2SnB3juukkUdMrwOpIkCO24Fkly9U0hvvvcfOqbQrx+42T6dc32OpIk\nEJWESBJzznHn64spqajmySuLGN4z1+tIkmC0uUkkiT3+jzLeWFjB988cwWmjenodRxKQSkIkSc0p\n3cq9by3nvLG9ubF4iNdxJEGpJESS0Jrttdzy4mcM75nLfd8ch5mm+5ZDo5IQSTKNwRBTX/gUn894\n4soistO161EOnV49Iknm/rdLKamo5okri+ifr08yyeHRSEIkiXy4chtPfLiG7xw/kDNGa0e1HD6V\nhEiS2FHTyO1/XsSwHp34yXmjvI4jSUKbm0SSgHOOH766mKr6Zp69dpLmZJIjRiMJkSTw3Nx1vL98\nK3edM5JRvXXqUTlyVBIiCW7t9lp+PXMZxSO6c/XkQq/jSJJRSYgkMOccd72+hDSfj99couMh5MhT\nSYgksJfnlfOvsh3cde4oeuVleh1HkpBKQiRBbalu4Fczl3HcoHwum9jf6ziSpFQSIgnqZ298TlMw\nzG++MQ6fT5uZpH2oJEQS0N+XbOLtki3cdsZwBhXkeB1HkphKQiTBVNU187PpJRzVtzPXnzTI6ziS\n5HQwnUiCeeDdUnbUNPLHqycS8Ot9nrQvvcJEEsiyTdU8N3cdVxw/kKP65nkdR1KASkIkQTjn+Pn0\nEvKy0rj9jOFex5EUoZIQSRBvLt7EJ2sq+f5ZI+iSne51HEkRKgmRBFDXFOTXM5cxpk9nLps4wOs4\nkkK041okATw8exWbqhp48PLx+HVMhHQgjSRE4tza7bU88Y81XDy+L0WF+V7HkRSjkhCJc7+euYw0\nv3HnOSO9jiIpSCUhEsfmra3knaVbuKF4CD07awI/6XgqCZE45Zzj1zOX0bNzBtedNNjrOJKiVBIi\ncertks18tn4Xt50+nKx0nY5UvKGSEIlDzaEw975VyrAenbj02H5ex5EUppIQiUMvzStnzfZafnT2\nSM3PJJ7Sq08kztQ0Bpn23gomDcrntFE9vI4jKU4H04nEmSf+Ucb2miaevGqUzlktntNIQiSObK9p\n5IkPyzhvbG+O6d/F6zgiKgmRePLonNU0NIe440zN8irxQSUhEic2VzXw3Nx1fGNCPwZ37+R1HBFA\nJSESNx6avZJw2HHLacO8jiKyh0pCJA6UV9bx8rxyvjWxP/3zs72OI7KHSkIkDjw4ayVmxtRTh3od\nRWQfKgkRj5Vtq+G1TzdyxXED6Z2X5XUckX2oJEQ8Nu39laT7fdxQPMTrKCJfoZIQ8dCKLbuZvqiC\nqyYX0j03w+s4Il+hkhDx0LT3V5Kd5ue7X9NU4BKfVBIiHlm1dTczl2ziysmFdM1J9zqOyAGpJEQ8\n8vDs1WQG/Fx/0iCvo4hEpZIQ8cDa7bW8sXAj3z5uAN06aV+ExC+VhIgHHpmzioDfxxTti5A4p5IQ\n6WDllXW8/ulGLp/Ynx6dM72OI9IqlYRIB3vsg9WYwXdP1nEREv9UEiIdaHNVA6/M38Clx/anTxcd\nXS3xTyUh0oEe+2A1Iee4UUdXS4JQSYh0kK27G3jxk/VcPL6vZnqVhNHmkjCzHDPzt0cYkWT25Idr\naA6FuekUzfQqieOgJWFmPjP7X2b2NzPbCiwHNpnZUjO7z8z0ihc5iMraJp6fu46vH92HQQU5XscR\niVksI4nZwBDgLqCXc66/c64HcBIwF7jXzK5ox4wiCe+p/1dGfXOIqRpFSIIJxLDO6c65ZjO7FFjy\nxULnXCXwGvCamaW1V0CRRBcMO575aB3nHtWbYT1zvY4j0iYHHUk455ojV58DXth7f4SZXbPfOq0y\ns7PNrNTMVpnZnQe4/2oz22ZmCyOX62P7MUTi1+aqBmoagzrrnCSktuy4Xg58wL4jh5tjfXCkXB4G\nzgFGA5eb2egDrPqyc+6YyOXJNuQTiTuhsGNzVT1njO7JqN6dvY4j0mZtKQnnnHsMeB2YbmZZgLXh\n8ZOAVc65MudcE/AScGEbHi+ScDZXNxAMO245dZjXUUQOSVtKYieAc+5Z4Cngb0BbPuzdFyjf6/aG\nyLL9fcPMFpvZq2bW/2BPWlpaSnFxMaWlpQDcf//9FBcXc//99+9zf3Fx8Z7HTJkyheLiYmbMmAHA\njBkzKC4uZsqUKXvW+eIxel4976E+7zXXXU/Z8hKywnWM7ZcX93n1vKn1vLGKuSScc6ftdf1V4AGg\nW8zfKTYzgELn3DjgXeCZA61kZlPMbL6ZzW9ujml3iEiHW7W1BocjP0dTgUviMudc6yuYmTvISjGu\ncwJwt3PurMjtuwCcc/dEWd8PVDrn8lp73qKiIjd//vzWVhHpcA3NIU66dzblz/2QUb07M2fOHK8j\niezDzBY454oOtl5Mx0mY2c1mNmC/b5BuZqea2TPAVTE8zzxgmJkNMrN04DJg+n7P2XuvmxcAy2J4\nXpG48+In69le00jfrprETxJbLMdJnA1cC7xoZoNp2TeRRUvBvAP83jn32cGexDkXNLOpwNuAH3ja\nOVdiZr8A5jvnpgO3mNkFQBCoBK4+hJ9JxFONwRD/54MyJhXmszVThxBJYjtoSTjnGoBHgEfMLBfI\nBeqcc7va+s2cczOBmfst+9le1++i5chukYT1yvwNbK5u4L5vjuOnL3mdRuTwxLzj2sxuAdYCnwD/\nMrOb2iuUSKJqCoZ5dM5qxg/owklDC7yOI3LYYpngb5qZXQX8JzDKOdcP+Bowxsx+2d4BRRLJXz7b\nwMZd9dxy2jDM2nIYkUh8assEfwXAR2b2KXAfsBq4zMy6tmM+kYTRHArz0OxVjOuXR/Hw7l7HETki\nYtkn8Vfgr2Z2PHAbsAkYBxwN5AOzzKyzc06n2pKU9sbCCsor6/n5+WM0ipCkEcunm75wE/BnYCEt\ns8GOApY454ojH2kVSVnBUJiHZ69idO/OnDaqh9dxRI6YthxxvRI4DngVyAQWAxdH7mtql3QiCeLN\nxZtYs71W+yIk6bRlJPFFGfwtchERWmZ6fXDWSkb2yuXM0T29jiNyRLX5HNcisq+ZSzaxelstN586\nDJ9PowhJLioJkcMQjowihvXoxDlH9fI6jsgRp5IQOQxvl2xmxZYapp46VKMISUoqCZFDFA47pr2/\nksEFOZw/ro/XcUTahUpC5BC9t2wLyzfvZuqpQ/FrFCFJSiUhcgicc/xh1koGdsvmgqM1ipDkpZIQ\nOQTvLN3C5xuruemUoQT8+jOS5KVXt0gbhcOOB95ZweCCHC4Zf6DTtIskD5WESBv9bckmSrfs5tbT\nh2kUIUlPr3CRNgiGwvzuvRWM6JnL1/WJJkkBKgmRNvjrwgrKttVy2xnDdVyEpASVhEiMmkNhpr2/\ngqP6duasMZqjSVKDSkIkRq/M30B5ZT13nDFCM71KylBJiMSgoTnEg7NWMmFAF4pH6KxzkjpUEiIx\neH7uOjZVNXDHmRpFSGpRSYgcRFV9Mw/NXsW/DSvgxKEFXscR6VAqCZGDeHTOaqrqm7nznJFeRxHp\ncCoJkVZU7Krn6X+u4aJj+jKmT57XcUQ6nEpCpBUPvLsCHNxx5nCvo4h4QiUhEsWyTdW89ukGrpo8\nkH5ds72OI+IJlYRIFL/5+3JyMwLcdMpQr6OIeEYlIXIA/1y1nQ9WbOOmU4bSJTvd6zginlFJiOwn\nFHb86m/L6JOXyVWTC72OI+IplYTIfl74ZD1LN1Vz17mjyEzzex1HxFMqCZG97Kxt4n/eKeX4wfmc\nP66313FEPKeSENnL/7xbSnV9M3dfMEbTb4igkhDZo6Siihc+Xs93jh/IyF6dvY4jEhdUEiKAc47/\nmr6ULtnp3H7GCK/jiMQNlYQIMH1RBZ+sreQHZ40gLzvN6zgicUMlISmvpjHIPTOXM7ZvHv9e1N/r\nOCJxJeB1ABGv/fat5WzZ3cAjV0zAr/NWi+xDIwlJafPWVvLc3HVcPbmQCQO6eh1HJO6oJCRlNTSH\n+NFri+mTl8X3z9TOapED0eYmSVkPzVpF2bZanr12EjkZ+lMQORCNJCQlLa2o5rEPVnPJhL58bXh3\nr+OIxC2VhKScYCjMj15bTF5WGj89b7TXcUTimsbYknIe/7CMJRurePDy8XTN0TTgIq3RSEJSyqLy\nXTzwzgrOHdtLE/iJxEAlISmjtjHIrS99Ro/cDO65eJwm8BOJgTY3Scq4e3oJ6yrreOk/jtfUGyIx\n0khCUsKMRRW8smADU08ZynGDu3kdRyRhqCQk6W3YWceP/7KEY/p34ZbThnkdRyShqCQkqTUFw9z6\n0kKcgz9cNp40v17yIm2hfRKS1O6eUcKCdTt58PLxDOiW7XUckYSjt1WStJ6fu44XPl7P904ewteP\n7uN1HJGEpJKQpPTJmkrunl5C8Yju/OAsTd4ncqhUEpJ0KnbVc+OfFjAgP5tpl43XOSJEDoNKQpJK\nfVOIKc/Np7E5zONXFpGXpeMhRA6HdlxL0giGwtz84qeUVFTz5JVFDO3RyetIIglPIwlJCs45fvrG\n57y3bCu/uGAMp43q6XUkkaSgkpCk8MC7K3jxk3JuOmUI3zmh0Os4IklDJSEJ75E5q3hw1ioun9Rf\npyEVOcJUEpLQnvywjN++VcpFx/Thvy8aq5ldRY4w7biWhPXonNXc+9Zyzh3bi/u/ebQ+6irSDlQS\nknCcc0x7fyW/f28lFxzdhwf+/WgCmpNJpF2oJCShhMKOn0//nOfnrufSY/tx7zfGaQQh0o5UEpIw\n6ptC3PbyQt4q2cx3Tx7MnWeP1D4IkXamkpCEsLmqgf94dj6fV1Tx0/NHc91Jg7yOJJISVBIS9+at\nrWTqC59S0xDkie8UcfpoHSgn0lFUEhK3wmHHox+s5oF3V9CvaxbPXDuJkb06ex1LJKWoJCQubdvd\nyO1/XsiHK7dz/rje3HPJWHIzNVmfSEdTSUjc+WDFNr7/yiKq65v59cVjuXxSf+2gFvGISkLixvaa\nRn755lLeWFjBkO45PHvtJEb11uYlES+pJMRzzjlemb+BX81cRl1TkFtPG8aNpwwhI+D3OppIylNJ\niKc+W7+Te2Yu55O1lUws7Mo9l4xlaI9cr2OJSIRKQjyxautu7nu7lLdLttAtJ517LhnLt4r649PR\n0yJxRSUhHWrN9loenbOKVxdsIDs9wO1nDOfakwbRKUMvRZF4pL9MaXfOOeat3ckTH5bx3rItpPl8\nXHPiIG4sHkK3ThlexxORVqgkpN3UNQV56/PNPPPRWhZtqKJLdho3FQ/lyskD6ZGb6XU8EYmBSkKO\nqC9GDa8uKGfmks3UNAYZVJDDLy86im9M6Et2ul5yIolEf7Fy2EJhx4J1O3l36WbeKtlMeWU92el+\nzhvbm0uP7cfEwnztkBZJUCoJOSQ7ahr5V9kO/rFiG+8v28qO2ibS/T5OGNKN204fztlH9dKoQSQJ\n6K9YYrK1uoHPynfxcVklH63ezvLNuwHIzQhwysgenDmmJycP7675lUSSjEpC9uGcY1tNI8s37WbZ\npmoWbdjFwvW7qKhqACAj4KOosCs/OGsEJwzpxri+eTp1qEgSU0mkqIbmEOWVdazbUcfaHbWs21HH\n6m01LN+8m8rapj3r9euaxYSBXbm2fxfGD+jCmD55ZKZpugyRVNGhJWFmZwPTAD/wpHPuN/vdnwE8\nCxwL7AC+5Zxb25EZk0FjMMT2miY2VzWwpbrlsrm6gS1VDWyqaqC8so5N1Q049+VjcjMDDO7eiTNG\n9WRk71xG9MplZK/O5Oeke/eDiIjnOqwkzMwPPAycAWwA5pnZdOfc0r1Wuw7Y6ZwbamaXAfcC3+qo\njPHAOUdDc5japiB1jSFqGoPUNQWpbQpR1xikJnLZWdfMrromdtY1s7O2iZ11Teyqa2ZnXRN1TaGv\nPG+630ePzhn06pzJ8YO7MbBbDoUF2QzslsPA/Gy6ZKdpOm4R+YqOHElMAlY558oAzOwl4EJg75K4\nELg7cv1V4CEzM+f2fs97ZGyqqmf9jjpCzhEKt1zCzhEMtXwNhSEYDu+5Hg47gmFHyLk918OR2188\n/otLMOxoCoZpCoVobA7TFAq33A62XG8M7nu7KRimMRiirjFEbVOQcAw/rRl0zkwjPyedLtlp9Oyc\nyYheuXTNTqdrdhrdOrUUwhfFkJ+TrhIQkTbryJLoC5TvdXsDcFy0dZxzQTOrAroB2490mOkLK7jn\n78uP9NPiMwj4fKQHWi4Zka/p/i+Xpft9dM5KI93/5f0ZAR/Z6QE6ZfjJzgiQk+4nOz1AToafnIzA\nl9fTA+RkBMjLSsOvYw9EpJ0l5I5rM5sCTAEYMGDAIT3HeeN6c1TfPPw+w+8zfGYE9rr+xXK/z/Cb\n4fe3fPX5Wkpg7+s+Hy3r+Ezv1kUkqXRkSWwE+u91u19k2YHW2WBmASCPlh3Y+3DOPQ48DlBUVHRI\nm6L6dc2mX9fsQ3moiEjK6MgPuM8DhpnZIDNLBy4Dpu+3znTgqsj1S4FZ7bE/QkREYtNhI4nIPoap\nwNu0fAT2aedciZn9ApjvnJsOPAU8Z2argEpaikRERDzSofsknHMzgZn7LfvZXtcbgG92ZCYREYlO\n8ymIiEhUKgkREYlKJSEiIlGpJEREJCqVhIiIRKWSEBGRqFQSIiISlUpCRESiUkmIiEhUKgkREYlK\nJSEiIlGpJEREJCqVhIiIRGWJfroGM9sGrPM6x34KaIdTrrYj5W1fiZQ3kbKC8h6Ogc657gdbKeFL\nIh6Z2XznXJHXOWKlvO0rkfImUlZQ3o6gzU0iIhKVSkJERKJSSbSPx70O0EbK274SKW8iZQXlbXfa\nJyEiIlFpJCEiIlGpJNqRmd1sZsvNrMTMfut1nliY2R1m5syswOssrTGz+yK/28Vm9hcz6+J1pv2Z\n2dlmVmpmq8zsTq/ztMbM+pvZbDNbGnm93up1poMxM7+ZfWZmb3qdJRZm1sXMXo28bpeZ2QleZ4qF\nSqKdmNkpwIXA0c65McD9Hkc6KDPrD5wJrPc6SwzeBY5yzo0DVgB3eZxnH2bmBx4GzgFGA5eb2Whv\nU7UqCNzhnBsNHA/cFOd5AW4Flnkdog2mAW8550YCR5Mg2VUS7ecG4DfOuUYA59xWj/PE4nfAD4G4\n31HlnHvHOReM3JwL9PMyzwFMAlY558qcc03AS7S8aYhLzrlNzrlPI9d30/IPrK+3qaIzs37AecCT\nXmeJhZnlAV8DngJwzjU553Z5myo2Kon2Mxz4NzP72Mw+MLOJXgdqjZldCGx0zi3yOsshuBb4u9ch\n9tMXKN/r9gbi+J/u3sysEBgPfOxtklb9npY3NGGvg8RoELAN+GNkE9mTZpbjdahYBLwOkMjM7D2g\n1wHu+gktv9t8WobuE4E/m9lg5+HHyQ6S98e0bGqKG63ldc69EVnnJ7RsKvlTR2ZLVmbWCXgN+E/n\nXLXXeQ7EzM4HtjrnFphZsdd5YhQAJgA3O+c+NrNpwJ3AT72NdXAqicPgnDs92n1mdgPweqQUPjGz\nMC3ztmzrqHz7i5bXzMbS8k5nkZlBy6abT81sknNucwdG3Edrv18AM7saOB84zcvyjWIj0H+v2/0i\ny+KWmaXRUhB/cs697nWeVpwIXGBm5wKZQGcze945d4XHuVqzAdjgnPtidPYqLSUR97S5qf38FTgF\nwMyGA+nEz8Re+3DOLXHO9XDOFTrnCml5QU/wsiAOxszOpmVzwwXOuTqv8xzAPGCYmQ0ys3TgMmC6\nx5mispZ3B08By5xzD3idpzXOubucc/0ir9XLgFlxXhBE/pbKzWxEZNFpwFIPI8VMI4n28zTwtJl9\nDjQBV8Xhu91E9hCQAbwbGf3Mdc59z9tIX3LOBc1sKvA24Aeeds6VeByrNScC3wGWmNnCyLIfO+dm\nepgp2dwM/CnypqEMuMbjPDHREdciIhKVNjeJiEhUKgkREYlKJSEiIlGpJEREJCqVhIiIRKWSEBGR\nqFQSIiISlUpC5Agzs++Z2cLIZY2ZzfY6k8ih0sF0Iu0kMhfSLOC3zrkZXucRORQaSYi0n2m0zCuk\ngpCEpbmbRNpBZIbagcBUj6OIHBZtbhI5wszsWOAZ4N+cczu9ziNyOLS5SeTIm0rLCadmR3ZeJ8Qp\nNkUORCMJERGJSiMJERGJSiUhIiJRqSRERCQqlYSIiESlkhARkahUEiIiEpVKQkREolJJiIhIVP8f\n3TLSVuCbSFQAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x114062dd0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(z, phi_z)\n",
    "plt.axvline(0., color='k')\n",
    "plt.axhspan(0., 1., facecolor='1.0', alpha=1., ls='dotted')\n",
    "plt.axhline(y=0.5, ls='dotted', color='k')\n",
    "plt.yticks([0., 0.5, 1.])\n",
    "plt.ylim(-0.1, 1.1)\n",
    "plt.xlabel('z')\n",
    "plt.ylabel('$\\phi (z)$')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "我们可以看到随着z趋向正无穷 $z \\to \\infty$，$\\phi(z)$无限接近于1；z趋向负无穷，$\\phi(z)$无限接近0.因此，对于sigmoid函数，其自变量取值范围为实数域，因变量值域为[0,1],并且sigmoid(0)=0.5。\n",
    "\n",
    "\n",
    "为了直观上对逻辑回归有更好的理解，我们可以和Adaline模型联系起来，二者的唯一区别是：**Adaline模型，激活函数$\\phi(z)=z$，在逻辑回归中，激活函数变成了sigmoid函数**。\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](https://ooo.0o0.ooo/2016/06/19/57676995058d2.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "由于sigmoid函数的输出是在[0,1]，所以可以赋予物理含义：样本属于正例的概率，$\\phi(z)=P(y=1|x;w)$。举例来说，如果$\\phi(z)=0.8$，意味着此样本是Iris-Versicolor花的概率是0.8，是Iris-Setosa花的概率是$P(y=0|x;w)=1-P(y=1|x;w)=0.2$。\n",
    "\n",
    "有了样本的预测概率，再得到样本的类别值就很简单了,和Adaline一样，使用单位阶跃函数：\n",
    "\n",
    "![](https://ooo.0o0.ooo/2016/06/19/57676accb1625.png)\n",
    "\n",
    "上式等价于：\n",
    "\n",
    "\n",
    "![](https://ooo.0o0.ooo/2016/06/19/57676b37b0235.png)\n",
    "\n",
    "\n",
    "逻辑回归之所以应用广泛，一大优点就是它不但能预测类别，还能输出具体的概率值，概率值很很多场景往往比单纯的类别值重要的多。比如在天气预测中下雨的可能性，病人患病的可能性等等。\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 学习逻辑斯底损失函数中的权重参数\n",
    "\n",
    "对逻辑回归模型有了基本认识后，我们回到机器学习的核心问题，怎样学习参数。还记得上一章Adaline中我们定义的差平方损失函数：\n",
    "\n",
    "$$J(\\mathrm{w})=\\sum_{i}\\frac{1}{2}(\\phi(z^{(i)}) - y^{(i)})^{2}$$\n",
    "\n",
    "\n",
    "我们求解损失函数最小时的权重参数，同样，对于逻辑回归，我们也需要定义损失函数，在这之前，先定义似然(likelihood)L的概念，假设训练集中样本独立，似然定义：\n",
    "\n",
    "$$L(\\mathrm{w})=P(\\mathrm{y}|\\mathrm{x};\\mathrm{w}) = \\prod_{i=1}^{n}P(y^{(i)}|x^{(i)}; \\mathrm{w})=(\\phi(z^{(i)}))^{y^{(i)}}(1-\\phi(z^{i}))^{1-y^{(i)}}$$\n",
    "\n",
    "与损失函数费尽全力找最小值相反，对于似然函数，我们要找的是最大值。实际上，对于似然的log值，是很容易找到最大值的，也就是最大化log-likelihood函数：\n",
    "$$l(\\mathrm{w})=log L(\\mathrm{w})=\\sum_{i=1}^{n}log(\\phi(z^{(i)})) + (1-y^{(i)})log(1-\\phi(z^{i}))$$\n",
    "\n",
    "\n",
    "接下来，我们可以运用梯度下降等优化算法来求解最大化log-likelihood时的参数。最大化和最小化本质上没有区别，所以我们还是将log-likelihood写成求最小值的损失函数形式：\n",
    "\n",
    "$$J(\\mathrm{w})=\\sum_{i=1}^{n}-log(\\phi(z^{(i)}))-(1-y^{(i)})log(1-\\phi(z^{(i)}))$$\n",
    "\n",
    "\n",
    "为了更好地理解此损失函数，假设现在训练集只有一个样本：\n",
    "\n",
    "$$J(\\phi(z),y;\\mathrm{w})=-ylog(\\phi(z))-(1-y)log(1-\\phi(z))$$\n",
    "\n",
    "上式等号右边的算式，如果y=0则第一项为0；如果y=1则第二项为0。\n",
    "\n",
    "![](https://ooo.0o0.ooo/2016/06/20/576794efcb16d.png)\n",
    "\n",
    "\n",
    "下图展示了一个训练样本时，不同$\\phi(z)$时对应的$J(w)$:\n",
    "\n",
    "\n",
    "![](https://ooo.0o0.ooo/2016/06/20/5767955034369.png)\n",
    "\n",
    "对于蓝线，如果逻辑回归预测结果正确，类别为1，则损失为0；对于绿线，如果逻辑回归预测正确，类别为0，则损失为0。\n",
    "如果预测错误，则损失趋向正无穷。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 调用scikit-learn训练逻辑回归模型\n",
    "\n",
    "\n",
    "如果我们自己实现逻辑回归，只需要将第二章中的Adaline中的损失函数替换掉即可，新的损失函数：\n",
    "\n",
    "$$J(\\mathrm{w})=-\\sum_{i}y^{(i)}log(\\phi(z^{(i)}))+(1-y^{(i)})log(1-\\phi(z^{i}))$$\n",
    "\n",
    "\n",
    "\n",
    "不过考虑到sklearn中提供了高度优化过的逻辑回归实现，同时也支持多类别分类，我们就不自己实现了，而是直接调用sklearn.linear_model.LogisticRegression，使用标准化后的Iris数据集训练模型：\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LogisticRegression(C=1000.0, class_weight=None, dual=False,\n",
       "          fit_intercept=True, intercept_scaling=1, max_iter=100,\n",
       "          multi_class='ovr', n_jobs=1, penalty='l2', random_state=0,\n",
       "          solver='liblinear', tol=0.0001, verbose=0, warm_start=False)"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr = LogisticRegression(C=1000., random_state=0)\n",
    "lr.fit(X_train_std, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAEKCAYAAAASByJ7AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XmcVPWZ7/HPlwZZRIQWBqMQiTFhREdBiRtq1EzcTW5c\n4mg0gzpjfMUk9HVJNN6bITrzyuLE6xqJJi4xLnGdGOMel7grICouRKNGXMEFARGE7uf+UaegaLqr\nD13LqeX7fr3q1VWnqs55qpV+6vk9v/M7igjMzMz6ZB2AmZnVBicEMzMDnBDMzCzhhGBmZoATgpmZ\nJZwQzMwMcEIwM7OEE4KZmQFOCGZmluibdQBrY7gUYzI47gy2ZfT4eRkc2cysdHNnzX03Ikb09Lq6\nSghjgOmZHHkGmhWc/f45mRzdzKwUba1tf0/zOg8ZpRSIKa1tWYdhZlYxTghraUprG22tU7IOw8ys\n7JwQeiGQk4KZNZy66iF0ZfmwYbw+dSpLN9sM+lQvvz3H7Sx8c4eqHa+3FvdbzJzWOSxvWZ51KGZW\n4+o+Ibw+dSrrbbcdY/r2RdU++Me5H++MH13tI6cSESx+fzG8D7NHzM46HDOrcXU/ZLR0s83YIItk\nUGDkrLkZHr17khjcOpjBywdnHYqZ1YG6Twj06ZNpMsibO2tk1iF0SaqF346Z1YP6Twg1YiLTa7ZS\nMDNLwwmhTG5/+GHGHnQQkz4zkTPaLs06HDOzteaEUAbt7e0c//Ofc9s55/DctdfywJ2XcP+N72Ud\nlpnZWqn7WUZr5ZvfhA8+WHP7sGHw29/2erePP/ssm40ezaajRgHwL1/+Mn+7/wI23fQCRo9/p9f7\nNTOrpuZKCB98kPvj39X2Erwxfz6jR65qKo8aOZLHZs9mItOZPmviyu1ODmZWyzxkVEW1OhPJzAyc\nEMpi4xEjmPvOqm//r7/zDhuPyK00O7HT+qxzZ410YjCzmuSEUAZfGDeOF197jVfeeINPli/nmrvu\n4iu77rry+c5JAZwYzKz2NFcPoUL69u3L+d//Pnt973u0t7dz9Fe+whaf/Wyq9+aTgvsLZpa15koI\nw4Z1P8uoRPtOmsS+kyb1+v1zZ410UjCzTDVXQihhammpJjKd6Uws+prCISQnBzOrNvcQqqirXkJ3\n3GMws2pzQqiytUkK4KmqZlY9zTVkVKc8lGRm1eAKIQNrWyUUcsVgZpWSWUKQNFrSvZKek/SspKa6\nSHGpScGJwczKLcshoxXAiRExU9J6wAxJd0XEcxnG1CtHn346tzz4IP8wbBizf//7qh3XQ0lmVk6Z\nVQgR8VZEzEzuLwKeBzau/HGLP+6Nyfvvz+3nnlv6jkrgisHMSlUTPQRJY4AJwGNdPHespOmSps8v\n8TgX3Tics64cuTIJRMBZV47kohuHl7TfXbfZhtYhQ9b6fROZXtLQUWceSjKzUmSeECQNBm4A2iJi\nYefnI+KiiJgYERNHlHCcCFi0pIWr72xdmRTOunIkV9/ZyqIlLWWpFGqFk4KZ9Uam004l9SOXDK6M\niBsreyw44Ru5cfar72zl6jtbAThsz/c54RvvkOW16NOcxby2VksKA18t677NrDFlOctIwG+A5yPi\nrOocc1VSyMs6GeSVc+iosw/mrv1wlpk1nyyHjCYBRwJ7SJqV3Pat5AHzw0SFCnsKjaytdQptrU01\ns9fM1lJmQ0YR8SBQte/mhT2D/DBR/jGUVikcdtpp3DdjBu8uWMCo/fbjx8ceyzFf/WoZoy+fwqRw\n9vvnZBiJmdWaplm6QoL1BrWv1jPIDx+tN6i9pGGjq//rv8oSYyV6CcW0tU5xUjCzlZomIQAce+C7\nRLDyj38+KdRCDyHPScHMstJUCQFY449/LSWDvCySQp6Tg1nzyvw8BOtaJWcdFePms1nzckKwLjkp\nmDWfphsyqifVHjrqzENJZs3FFYKl4orBrPE5IZTB3LffZvfjjmPc17/OFl//OudcfXXZ9p1VL6Er\n7i+YNTYnhDLo27cvv2hr47lrr+XRSy/lguuv57mXXy7b/mspKYCrBbNG1XQ9hFcHvs+MoW+ysN9S\nhiwfwLYLNmLMx60l7fNTw4fzqeG5JbTXW3ddNh8zhjfmz2fcppuWI+Sa5P6CWeNpqgrh1YHvc++I\nl/moZRn921v4qGUZ9454mVcHvl++Y7z5Jk/OmcP2W2xRtn1C7VUJhTyUZNYYuq0QJB2Y4v1LI+LW\nMsZTUTOGvklLh+gXLQC5nx3tzBj6ZslVAsDiJUs46Ac/4OwTTmDI4MEl76+zrGcd9SSfFFwxmNWn\nYkNGFwN/oPgCdLsCdZMQFvZbSv/2ltW29Y0+LOy3tOR9L1+xgoN+8AO+sffeHLjHHiXvr545MZjV\np2IJ4baIOLrYmyX9rszxVNSQ5QP4qGXZygoBYIU6GLJ8QEn7jQiOOeMMNh8zhhO+8Y1Swyyq1quE\nQl4nyay+dNtDiIgjenpzmtfUkm0XbER7n2C52glyP9v7BNsu2Kik/T701FNcceut3DN9OuMPP5zx\nhx/OrQ89VKao65v7C2b1o9c9hEpf8rISxnzcyu7zWX2W0XulzzLaefx44oknyhRlz+qpSsjzrCSz\n2ldsyOiA5Oc/ADsB9ySPdwceBuouIUAuKZSjgZy1ekwKeR5KMqtNxYaMjoqIo4B+wLiIOCgiDgK2\nSLZZxmp5KmpPPJRkVnvSnIcwOiLeKnj8DvDpCsVjTcaJwax2pDlT+c+S7gDyC/QcCtxduZCsGXmq\nqln2eqwQIuI7wDRg6+R2UUR8t9KBWTr1PGzUFVcLZtlJu5bRTGBRRNwtaZCk9SJiUSUDs/TqucHc\nFVcLZtnosUKQ9O/A9cCvkk0bA/9TyaDqyYJFi/jlddf1+v1nX3UVS5aWfqb0jBn38dRTD5e8n1ri\n/oJZdaVpKh8PTAIWAkTEi+SmohpJQrj++l6//+xrrilLQnhrxlV88PTvS95PLXJiMKuONENGyyLi\nEym3pJGkvkBUNKo6csr55/O3N95g/OGH8+Xtt+fMKVM484oruPauu1i2fDlf2203fvytb/HRxx/z\n9VNP5fV582hvb+f/HnMM77z/Pm/On8/uxx3H8KFDuXfatNX3fd553PzAA/RtaWHP7bfnv9vamP/B\nBxz3k5/w2ttvA3D2iSey8YgRTLvhBlpaWrjotvs5+eTzmDBhlyx+HRXlk9vMKitNQrhf0g+BgZK+\nDHwb+GNlw6ofP/3Od5j9t78x66qrALjz0Ud58bXXePzyy4kIvnLiifxl5kzmL1jARsOH86ezzwbg\nw8WLWX/wYM666irunTaN4UOHrrbf9xYs4Kb77uOF669HEgsW5Vo2U37xC/734Yez8/jxvPb22+z1\n3e/y/HXXcdxBBzF44EBOOvLIhuondMcnt5mVX5qEcApwDPAM8C3g1oi4uKJR1bE7H32UOx97jAnJ\nIneLP/6YF+fOZZfx4znx7LP5wXnnsf/OO7PLhAlF97P+4MEM6N+fY844g/133pn9d8l947/78cdX\nuxrbwo8+YvGSJZX7QDXMzWez8kqTEL4BXFOYBCTtHxG3VC6s+hURnDp5Mt86cM2loGZecQW3PvQQ\n/+fCC/nSF77Aj/7937vdT9++fXn8ssv48xNPcP2f/8z5113HPRdeSEdHB49eeikD+vfv9r2NNuuo\nJ64WzMojTVP5POABSZsXbDu9QvHUnfUGDWJRwTf0vXbckUtuvnnlt/Y35s1jXtIrGDRgAEfsuy8n\nH3kkM+fMWfX+jz5aY7+Llyzhw8WL2XfSJP7fCSfw1IsvArDnDjtw3rXXrnzdrML9NGmlAKsaz24+\nm/VemgrhFXJDRtdLmhoR11H8ojlNZYOhQ5m09dZseeih7LPTTpw5ZQrPv/IKOx6du5TE4EGD+N3p\np/PS3LmcfO659JHo17cvF55yCgDHfu1r7P2977HRiBGrNZUXLVnCV088kaWffEJEcFZbGwDnnnQS\nx//sZ2x12GGsaG9n1wkTmHbqqRywyy4cfMop/OH++znv5JPZZQJNVSUUcsVg1juKKD5hSNLMiNhG\n0nByy1c8BewZEVtVI8BCE6XofF7u87fdxubJBe5tTdOZyLvvPs8++2ze84sbkBODGbS1ts2IiB6/\nIaapEN4CiIh3Je0F/AzYssT4rEomMp3bWTfrMDLjqapm6aVZy2i/gvsdEXFyRKTpPZjVFPcXzIor\ndsW0syOiTdIf6eJEtIj4SkUjS6ujg8BNje4EsEnHK0BzDhl15qmqZt0rNmR0RfLzvyt1cEmXAPsD\n8yKiV8NQA156ifdaW9mgb18nhU4CeG/FCga89FLWodQcJwazNfXYVK7owaVdgcXAb9MkhK6aysuH\nDeP1qVNZutlm0McjWavp6GDASy8xaupU+n3wAfKKI91yYrBGlrap3G1CkPQMRdYsKtcsI0ljgFt6\nmxBs7TgpdM9JwRpVOWYZ7Z/8PD75mR9COoIqLm4n6VjgWPB1O8shkJNCNzwjyZpdmvMQnoyICZ22\nzYyIbcoSgCuEqnNCSM+JwRpB2gohzaC7JE0qeLBTyvdZjQq331PzVFVrJmlOTDsauFTS+snjBck2\ns6bgoSRrFkW/6UvqA2wWEVsDWwNbR8T4iJhZjoNLuhp4BBgr6XVJx5Rjv9YzVwm944rBGlnRCiEi\nOiR9H7g2Ij4s98Ej4rBy79Os0nwOgzWqNL2AuyWdJGm0pNb8reKRWcW5SiiNqwVrNGl6CIcmP48v\n2BbApuUPx6rN01BLk2V/IQKk7h+bra00i9t9poubk0EDcaVQHtWsGB6+bEteeXQk+VnjEdCxQnR0\nVC0Ea0BpKgQkbQmMAwbkt0XEbysVlFm9qkbFEAEbjp3PJl+YR0c79GnJbe/TN4gOVwrWez0mBEn/\nAexGLiHcCuwDPAg4ITQQDx2VX6WazxJ8Zod36GiHjhUt9OnTvnK5X/VxMrDeS9NUPhj4EvB2RBxF\nbvrp+sXfYmZ5lRhKklZVBoUjfk4GVoo0CeHjiOgAVkgaAswDRlc2LMuCewmV09Y6payJIQI62vMP\nVt9u1ltpEsJ0SUOBi4EZwExyJ5NZA3JSqKxyJIYIeOXRkYDo07d9tQoh30PonBicKCyNHnsIEfHt\n5O40SbcDQyLi6cqGZdbYSukvSPD2nBFArpcgrZplpJZgxSd9mDtzxGrPLXh9MAOHLmPAesvL+jms\nsXRbIUjapvMNaAX6JvfNrES9rRZ2mjx75R98SHoKfQMJHrx4a246bXfuu2ACEXDfBRO45F/355HL\nt3SlYEUVu0DOvcndAcBE4ClyxelWwPSI2LEqERbw8tfV5VlH1fXt/7mBz+/6esn7ySeBmTeOXblt\nmwPnsNvxT7rp3KRKXv46InaPiN2Bt4BtImJiRGwLTADeKF+oZgawwSYLee3JEbz25D+UtB8Jdjv+\nydW2ORlYGmmaymMj4pn8g4iYDWxeuZCsVrjBXF3v/X0I6wxcAbAyKUSRGUTdDf/kK4TOjwvPajbr\nSpqE8LSkX0vaLbldDLip3CScFKrnnnO3pW//dtYZuBwI3n5hGO+9OoRpB3+Vhy/bco0/6vddMIGH\nL1v9QoOFw0XbHDiHHb45m/U3XMyMG8Zy3wUT6Ojo+n1mkC4hHAU8C0xJbs8l28ysjF64ZwzXn7w7\nC+ety7qty1g4b12uP3l3XrhnDNee8CVm3jh2tUbxzBvHsmxxv9W+8UvQf/BytjlwDl/89pN8srgf\nH749mPU3XMw66y7n/l92/T4zSHFN5VripnJ23GCuHaPHvwMUbxTn1zPKJ48ZN4xd+To3mJtP2a6p\nLGmSpLsk/VXSy/lbecK0euGho9oxd9ZI5s4ayR9+tGu3f9QLp6N2/uPvZGDdSTNk9BvgLGBn4AsF\nNzPLWE/nMXRuMMPqDWazQmkSwocRcVtEzIuI9/K3ikdmNcdVQm1qa52yxnUQOjrWbDCf8Odr2ObA\nOav1IswK9dhDkPRToAW4EViW3x4RMysb2prcQ6gN7ifUrrPfP4eODrjj59sx/n+9yLCNF7N8aR8G\nD1+2sqfw9B835aP3B7HT5NlZh2tVkraHkOYCOdsnPwt3FsAevQnMzCqnrXUK/7jHqxx85r209Gun\n/+Dl9B+8ek9hqwNedg/BupRmcbvdqxGI1Q9fTKe27fG9GaxY1sKSBQMYNuqjNZ53MrDupL2E5n7A\nFqx+Cc3TKxWUmfXeBpss5KP3+wOwYlkf2ttF/0HtPbzLLN2002nAocB3yS1udwiwSYXjshrnBnPt\nKlwCY/knLagPLP+4haWL+mUcmdW6NLOMdoqIbwIfRMSPgR2Bz1c2LKsHTgq1qXAJjEXvDKRPn0At\nHSycN5C3XxjGu68MyTpEq1GpLqGZ/FwiaSNgOfCpyoVkZqUoXAKjpR+8PWcY78zZgJa+rFwOoxLX\nebb6l6aHcEtyCc0zyV0+M4BfVzQqqxtuMNemF+4Zwwv3jCn6msKk0Jsrt1njSXMeQv+IWJa/T66x\nvDS/rZp8HkLtclJoDE4MjalsaxkBj+TvRMSyiPiwcJuZNQ4PJTW3boeMJG0IbAwMlDQBVnYQhwCD\nqhCb1REPHTUODyU1r2I9hL2AycAo4BesSgiLgB9WNiwzqwVtrVOcFJpItwkhIi4HLpd0UETcUMWY\nrE65SmhM+YrBiaHxpekhjJI0RDm/ljRT0p4Vj8zqks9NaFxtrVPcY2hwaRLC0RGxENgT2AA4Evhp\nOQ4uaW9JcyS9JOmUcuzTsuek0NicGBpXmoSQ/9e9L/DbiHi2YFuvSWoBLgD2AcYBh0kaV+p+zaw6\nnBgaT5qEMEPSneQSwh2S1gM6enhPGtsBL0XEyxHxCXAN8NUy7NdqgKuE5uHE0DjSnKl8DDAeeDki\nlkjaADiqDMfeGJhb8Ph1Vl17wczqjJvP9a/bCiE5D4GI6IiImRGxIHn8XkQ8XfiaSpJ0rKTpkqbP\nr/TBrKxcJTQnVwv1q1iFcCuwTQ/vT/Oa7rwBjC54PCrZtpqIuAi4CHJLV/TyWGZWRT65rT4V6yFs\nLWlhkdsiYGQJx34C+Jykz0haB/gX4OYS9mc1KJArhSbniqF+FDsxraWSB46IFZK+A9wBtACXJDOY\nzKzBuL9QH9LMMqqYiLg1Ij4fEZ+NiP/KMharLFcJBq4Wal2qayqbmZWL+wu1K9MKwZqLqwTrzOcw\n1JZUCUFSi6SNJH06f6t0YGbWPJwUakOPQ0aSvgv8B/AOq85QDmCrCsZlDSpfJXhVVOvMQ0nZS1Mh\nTAHGRsQWEfFPyc3JwMwqxhVDNtIkhLnAh5UOxJqL+wnWE/cXqq/YJTRPSO6+DNwn6U/AsvzzEXFW\nhWMzM/NQUhUVqxDWS26vAXcB6xRsG1z50KzRuUqwteWKobKKnan8YwBJh0TEdYXPSTqk0oGZmXXF\nZz1XTpoewqkpt5mtNVcJ1lvuMZRfsR7CPuQuirOxpHMLnhoCrKh0YNY8AnkaqvWaK4byKVYhvAnM\nAJYmP/O3m4G9Kh+aNRNXClYqVwulK9ZDeAp4StKVEbG8ijGZmfWKq4XSFBsyeobcGclIa35788lp\nVm4eOrJycWLonWJLV+yf/Dw++XlF8vMI8L9aM6t9TgxrRxHF/7ZLejIiJnTaNjMienvpzF6bKMX0\nah/Uqs5VglVSMyaHtta2GRExsafXpZl2KkmTCh7slPJ9ZmY1x83n7qW5QM4xwCWS1gcEfAAcXdGo\nrKm5l2CV5qGkrvX4TT8iZkTE1sDWwFYRMT4iZlY+NGtmnoZq1eBqYXXFZhkdERG/K1jkLr8d8OJ2\nVnmuFKwavHjeKsUqhHWTn+t1czMzayjNXjEUOzHtV8ndn0XE0irFY7YaVwlWbc3cX0jTVJ4t6R3g\ngeT2YET4gjlm1tCacSgpTVN5M+Aw4BlgP3LLWcyqdGBmeW4wW9aaZSipxwpB0ihgErALuZlGzwIP\nVjgus9V46Miy1gxDSWlOMHsNaANui4gdI2K/iPhJheMyW4MrBasFjVwtpOkhTAB2Bg6XdArwInB/\nRPymopGZdcGVgtWCRu0vpOkhPAVcDlwK3AN8EfhRheMyM6sLjVQxpOkhTAf6Aw+Tm2W0a0T8vdKB\nmZnVi0apGNIMGe0TEfMrHolZSh42slpWz83nNENGTgZWc9xgtlpXj0NJXsbazKxC2lqn1FVicEKw\nuuUqwepFvSSGYqudHljsjRFxY/nDMVs77idYPan1/kKxpvIBRZ4LoNcJQdIhwFRgc2C7CF8Z08ya\nR1vrlJpMCsVWOz2qgsedDRwI/KqnF5r1xFWC1aNanKqaZtopkvYDtgAG5LdFxOm9PWhEPJ/st7e7\nMFuNk4LVs1qpGHpsKkuaBhwKfJfcNZUPATapcFxmZk2lFhrPaSqEnSJiK0lPR8SPJf0CuK2nN0m6\nG9iwi6dOi4g/pA1Q0rHAsQCfTvsma0quEqwRZDmUlCYhfJz8XCJpI+A94FM9vSki/rmUwAr2cxFw\nEcBEyf/azaxpVHsoKU1CuEXSUOBMYCa5GUa/rmhUZr3kKsEaTTWnqiqi+D8eSf0jYln+PrnG8tL8\ntl4dVPoacB4wAlgAzIqIvXp630TJ81MtFScFa1S9SQxtrW0zImJiT69Lc6byI/k7EbEsuZ7yI0Ve\n36OIuCkiRkVE/4gYmSYZmJlZZZvP3SYESRtK2hYYKGmCpG2S227AoIpEY1YmXtbCGl0lkkKxHsJe\nwGRgFHBWwfaFwA/LHolZmbmfYI2u3DOSip2pfDlwuaSDIuKGko9kZmYVU47mc5oewkOSfiPpNgBJ\n4yQd0+sjmlWRh46s2ZQylJQmIVwK3AFslDz+K9DW6yOamVlF9bbxnGba6RMR8QVJT0bEhGTbrIgY\n37tQe8/TTq233Euw5qayTTv9SNIG5E5IQ9IOwIclRmdWVR46MutZmjOVTwBuBj4r6SFyJ5MdXNGo\nzCrAs47MiusxIUTETElfBMaSW+10TkQsr3hkZmZWVT0mBEkDgG8DO5MbNnpA0rSIWFrp4MzKzVWC\nWffSDBn9FlhEbu0hgMOBK8hdF8HMzBpEmoSwZUSMK3h8r6TnKhWQWaW5SjDrWppZRjOTmUUASNoe\n8OxPq2uedWS2pjQVwrbAw5JeSx5/Gpgj6RkgImKrikVnVkGuFMxWlyYh7F3xKMzMLHNppp3+vRqB\nmGXBVYLZKml6CGZm1gScEKzpucFsluOEYGZmgBOCGeAqwQycEMxWclKwZueEYFbAScGamROCmZkB\nTghma3CVYM3KCcHMzAAnBLMuuUqwZuSEYNYNJwVrNk4IZkU4KVgzcUIwMzPACcGsR64SrFk4IZiZ\nGeCEYJaKqwRrBk4IZmYGOCGYpeYqwRpdJglB0pmSXpD0tKSbJA3NIg6zteWkYI0sqwrhLmDLiNgK\n+CtwakZxmK01JwVrVJkkhIi4MyJWJA8fBUZlEYdZbzkpWCOqhR7C0cBtWQdhZtbs+lZqx5LuBjbs\n4qnTIuIPyWtOA1YAVxbZz7HAsQCfrkCcZmaWo4jI5sDSZOBbwJciYkma90yUYnpFozJbOyKbfz9m\na0czImJiT6+qWIVQjKS9ge8DX0ybDMxqUSAnBWsYWfUQzgfWA+6SNEvStIziMCuZG8zWKDKpECJi\nsyyOa1YprhSsEdTCLCMzM6sBTghmZgY4IZiVjXsJVu+cEMzKyEnB6pkTgpmZAU4IZmXnKsHqlROC\nWQU4KVg9ckIwMzPACcGsYlwlWL1xQjAzM8AJwayiXCVYPXFCMKswJwWrF04IZmYGOCGYVUUgVwpW\n8zK7YlpvSJoP/D3rOAoMB97NOogMNfPnb+bPDs39+evxs28SESN6elFdJYRaI2l6msvSNapm/vzN\n/NmhuT9/I392DxmZmRnghGBmZgknhNJclHUAGWvmz9/Mnx2a+/M37Gd3D8HMzABXCGZmlnBCKJGk\nMyW9IOlpSTdJGpp1TNUk6RBJz0rqkNSQMy86k7S3pDmSXpJ0StbxVJOkSyTNkzQ761iqTdJoSfdK\nei75f35K1jGVmxNC6e4CtoyIrYC/AqdmHE+1zQYOBP6SdSDVIKkFuADYBxgHHCZpXLZRVdVlwN5Z\nB5GRFcCJETEO2AE4vtH+2zshlCgi7oyIFcnDR4FRWcZTbRHxfETMyTqOKtoOeCkiXo6IT4BrgK9m\nHFPVRMRfgPezjiMLEfFWRMxM7i8Cngc2zjaq8nJCKK+jgduyDsIqamNgbsHj12mwPwrWM0ljgAnA\nY9lGUl59sw6gHki6G9iwi6dOi4g/JK85jVxJeWU1Y6uGNJ/frFlIGgzcALRFxMKs4yknJ4QUIuKf\niz0vaTKwP/ClaMB5vD19/ibzBjC64PGoZJs1AUn9yCWDKyPixqzjKTcPGZVI0t7A94GvRMSSrOOx\ninsC+Jykz0haB/gX4OaMY7IqkCTgN8DzEXFW1vFUghNC6c4H1gPukjRL0rSsA6omSV+T9DqwI/An\nSXdkHVMlJRMIvgPcQa6peG1EPJttVNUj6WrgEWCspNclHZN1TFU0CTgS2CP5tz5L0r5ZB1VOPlPZ\nzMwAVwhmZpZwQjAzM8AJwczMEk4IZmYGOCGYmVnCCcGqRtJkSRuleN1lkg5Ou70Mcf2w4P6YNCt5\nJrG8Ium4Iq8ZX85picnv7/wS93FfflVaSbeWujqvpN0k3ZLcPzRZAfaWUvZp2XFCsGqaDPSYEDLw\nw55f0qWTI6LYeSfjgczmqUsquhJBROwbEQvKdbyI+D3wb+Xan1WfE4L1SvJN+gVJV0p6XtL1kgYl\nz20r6X5JMyTdIelTyTf7icCVyQk9AyX9SNITkmZLuig5EzTt8dc4RrL9Pkk/k/S4pL9K2iXZPkjS\ntcla9jdJekzSREk/BQYmMeXXoWqRdHGy5v2dkgamiOeQ5HM8JekvyVnMpwOHJvs+VNJ2kh6R9KSk\nhyWNTd47WdKNkm6X9KKknxfs96jkczxO7sSo/PYDks/wpKS7JY1Mtk+VdIWkh4Arkt/zNcl/o5uA\ngQX7eFXScEnHFZxo9Yqke5Pn90zinSnpOuXW8MlfD+IFSTPJLX1ujSIifPNtrW/AGCCAScnjS4CT\ngH7Aw8AC6GtnAAADuUlEQVSIZPuhwCXJ/fuAiQX7aC24fwVwQHL/MuDgLo55GXBwimP8Irm/L3B3\ncv8k4FfJ/S3JLUQ4MXm8uNPnWgGMTx5fCxzRXSwFj58BNk7uD01+TgbOL3jNEKBvcv+fgRsKXvcy\nsD4wAPg7ufWSPgW8BowA1gEeyu8PGMaqE0v/reAzTwVmAAOTxycU/G626vS5XwWGF8TXD3gAOAAY\nTu4aF+smz/0A+FES31zgc4CS388tBfvYrfCxb/V18+J2Voq5EfFQcv93wPeA28n9wb0r+cLfArzV\nzft3l/R9YBDQCjwL/DHFccf2cIz8omMzyP2BB9gZOAcgImZLerrI/l+JiFld7KOYh4DLJF1bcPzO\n1gcul/Q5csm0X8Fzf46IDwEkPQdsQu6P8n0RMT/Z/nvg88nrRwG/TyqjdYBXCvZ1c0R8nNzfFTgX\nICKe7uFznwPcExF/lLQ/uQsAPZT8jtcht2TFP5L7/byYxPQ74Ngi+7Q64oRgpei87kmQ+9b4bETs\nWOyNkgYAvyT3bXWupKnkvn2m0dMxliU/2+nd/+PLCu63UzDM0p2IOE7S9sB+wAxJ23bxsjOAeyPi\na8qtp39fkWP2FPd5wFkRcbOk3chVBnkf9RRvZ8qt2LsJuXWaIPc7visiDuv0uvFru2+rH+4hWCk+\nLSn/R/lw4EFgDjAiv11SP0lbJK9ZRG4hQFj1x//dZGx6bWYPFTtGdx4Cvp68fhzwTwXPLVduWeNe\nk/TZiHgsIn4EzCc35FP4eSFXIeSXyp6cYrePAV+UtEES3yHd7Otfi+zjL+T+2yBpS3LDRp1j35bc\nkNoREdGRbH4UmCRps+Q160r6PPACMEbSZ5PXHdZ5f1a/nBCsFHPIXVf2eXJj2hdG7rKSBwM/k/QU\nMAvYKXn9ZcA0SbPIfSO+mNw1me8gt6x0Kj0cozu/JJdEngP+k9zw1IfJcxcBTxc0lXvjTEnPKDdl\n9WHgKeBeYFy+qQz8HPiJpCdJUblExFvkvvk/Qi6hPV/w9FTgOkkzgHeL7OZCYHDy3+h0ckNgnX2H\n3JDdvUmsv06GqSYDVyfDTI8A/xgRS8kNEf0paSrP6+lzWP3waqfWK8mQxy0RsWXGoaQiqQXoFxFL\nk2+3dwNjk+TSm/1dRu7zX1/GMOteMnx1UkTsn3UstvbcQ7BmMYjcN+B+5MbHv93bZJD4EDhD0vAo\nfi5C00iqoP+g6yrE6oArBDMzA9xDMDOzhBOCmZkBTghmZpZwQjAzM8AJwczMEk4IZmYGwP8HoDaP\nMoXXlJIAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x114010d90>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_decision_regions(X_combined_std, y_combined, classifier=lr, test_idx=range(105, 150))\n",
    "plt.xlabel('petal length [standardized]')\n",
    "plt.ylabel('petal width [standardized]')\n",
    "plt.legend(loc='upper left')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "细心的你一定会问在初始化LogisticRegression时那个参数C是什么意思呀？在下一节讲正则化的时候我们再好好讨论讨论。\n",
    "\n",
    "有了逻辑回归模型，我们就可以预测了，如果你想知道输出概率，调用predict_proba方法即可，"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/lj/.pyenv/versions/2.7.12/lib/python2.7/site-packages/sklearn/utils/validation.py:395: DeprecationWarning: Passing 1d arrays as data is deprecated in 0.17 and will raise ValueError in 0.19. Reshape your data either using X.reshape(-1, 1) if your data has a single feature or X.reshape(1, -1) if it contains a single sample.\n",
      "  DeprecationWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([[2.05743774e-11, 6.31620264e-02, 9.36837974e-01]])"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr.predict_proba(X_test_std[0, :])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "\n",
    "上面的array表示lr认为测试样本属于Iris-Virginica类别的概率为93.683%，属于Iris-Versicolor的概率为6.316%。\n",
    "\n",
    "\n",
    "不论Adaline还是逻辑回归，使用梯度下降算法更新权重参数时，用到的算式都一样，$\\Delta w_{j} = -\\eta \\frac{\\partial J}{\\partial w_{j}}=\\eta \\sum_{i}(y^{(i)}-\\phi(z^{(i)}))x_{j}^{(i)} $。\n",
    "\n",
    "我们好好推导一下这个计算过程，首先计算log-likelihood函数对权重参数的偏导数：\n",
    "\n",
    "\n",
    "![](https://ooo.0o0.ooo/2016/06/20/57679b382ba9d.png)\n",
    "\n",
    "然后，计算sigmoid函数的导数：\n",
    "\n",
    "![](https://ooo.0o0.ooo/2016/06/20/57679bd85ae8c.png)\n",
    "\n",
    "将上式代入到损失函数偏导数：\n",
    "![](https://ooo.0o0.ooo/2016/06/20/57679c5651e00.png)\n",
    "\n",
    "我们的目标是求解使log-likelihood最大值时的参数w，因此，对于参数更新，我们按照：\n",
    "\n",
    "![](https://ooo.0o0.ooo/2016/06/20/57679cb31605b.png)\n",
    "\n",
    "\n",
    "由于每次更新权重时，对于所有参数同时更新，所以可以写成向量形式：\n",
    "\n",
    "![](https://ooo.0o0.ooo/2016/06/20/57679cf9bbfa9.png)\n",
    "\n",
    "我们定义$\\Delta w$\n",
    "\n",
    "![](https://ooo.0o0.ooo/2016/06/20/57679d60b0989.png)\n",
    "\n",
    "由于最大化log-likelihood等价于最小化损失函数$J$,我们可以将梯度下降算法的权重更新写作：\n",
    "\n",
    "\n",
    "![](https://ooo.0o0.ooo/2016/06/20/57679da3c5e99.png)\n",
    "\n",
    "这和Adaline中的权重更新算式完全相同。\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 使用正则化解决过拟\n",
    "\n",
    "过拟合(overfitting)是机器学习中很常见的问题，指的是一个模型在训练集上表现很好但是泛化能力巨差(在测试集上的表现糟糕)。如果一个模型饱受过拟合困扰，我们也说此模型方差过高，造成这个结果的原因可能是模型含有太多参数导致模型过于复杂。同样，模型也可能遇到欠拟合(underfitting)问题，我们也说此模型偏差过高，原因是模型过于简单不能学习到训练集中数据存在的模式，同样对于测试集表现很差。\n",
    "\n",
    "\n",
    "虽然到目前为止我们仅学习了用于分类任务的几种线性模型，过拟合和欠拟合问题可以用一个非线性决策界很好的演示：\n",
    "\n",
    "![](https://camo.githubusercontent.com/c07390d7a7cc092a8f65feab1179bee0e93d85f9/68747470733a2f2f6f6f6f2e306f302e6f6f6f2f323031362f30362f32302f353736376137323531666130382e706e67)\n",
    "\n",
    "\n",
    "怎样找到bias-variance之间的平衡，常用的方法是正则化(regularization)。正则化是解决特征共线性、过滤数据中噪音和防止过拟合的有用方法。正则化背后的原理是引入额外的信息(偏差)来惩罚过大的权重参数。最常见的形式就是所谓的L2正则(L2 regularization,有时也被称为权重衰减，L2收缩)：\n",
    "\n",
    "$$\\frac{\\lambda}{2}||\\mathrm{w}||^{2}=\\frac{\\lambda}{2}\\sum_{j=1}^{m}w_{j}^{2}$$\n",
    "\n",
    "此处的$\\lambda$就是正则化系数。\n",
    "\n",
    "\n",
    "**Note** 正则化是为什么特征缩放如此重要的另一个原因。为了正则化起到作用，我们需要保证所有的特征都在可比较范围(comparable scales)。\n",
    "\n",
    "\n",
    "如何应用正则化呢？我们只需要在现有损失函数基础上添加正则项即可，比如对于逻辑回归模型，带有L2正则项的的损失函数：\n",
    "\n",
    "\n",
    "![](https://ooo.0o0.ooo/2016/06/20/5767a9baccf68.png)\n",
    "\n",
    "\n",
    "通过正则系数$\\lambda$,我们可以控制在训练过程中使得参数$w$比较小。$\\lambda$值越大，正则化威力越强大。\n",
    "\n",
    "现在我们可以解释LogisticRegression中的参数C:\n",
    "\n",
    "$$C=\\frac{1}{\\lambda}$$\n",
    "\n",
    "所以，我们可以将逻辑回归 正则化的损失函数重写为\n",
    "\n",
    "![](https://ooo.0o0.ooo/2016/06/20/5767aabb83aaf.png)\n",
    "\n",
    "\n",
    "如果我们减小C的值，也就是增大正则系数$\\lambda$的值，正则化项的威力也增强。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "weights, params = [], []"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#  我们训练了十个带有不同C值的逻辑回归模型。\n",
    "for c in np.arange(-5, 5, dtype=float):\n",
    "    lr = LogisticRegression(C=10**c, random_state=0)\n",
    "    lr.fit(X_train_std, y_train)\n",
    "    weights.append(lr.coef_[1])\n",
    "    params.append(10**c)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "weights = np.array(weights)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAEOCAYAAACJlmBtAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl8VNXdx/HPL3tCWMK+E1RkixAgiIpa12IVN0pd8RGx\noFXU1qdP7aOtW6tVq11QeSpURCogilqqxWptbRUVJcGgLOJWlgCyBxKSkGXO88cdIGEJA8nkzvJ9\nv17zmpl77tz74zC5vzn3nHuuOecQERFJ8DsAERGJDEoIIiICKCGIiEiQEoKIiABKCCIiEqSEICIi\ngBKCiIgEKSGIiAighCAiIkFKCCIiAkCS3wEcibZt27rs7Gy/wxARiSoFBQVbnHPtDrdeVCWE7Oxs\n8vPz/Q5DRCSqmNnqUNbTKSMREQGUEEREJEgJQUREgCjrQziYqqoqioqKqKio8DuUuJOWlkbXrl1J\nTk72OxQRaQRRnxCKiopo3rw52dnZmJnf4cQN5xxbt26lqKiInj17+h2OiDSCqD9lVFFRQZs2bZQM\nmpiZ0aZNG7XMRGJI1LcQACUDn6jeJRbVBBzlVTWU7a6mrLKG6oADHM6BAwIu+NqB4yCv8VrQe5/3\nLvPeB4LrUmt5oPb6ECxzBAL7tje4RxZtM1PD+m+PiYQQbaZPn863v/1tOnfuXO96Y8eOZeTIkYwe\nPTqk5Q314IMPcueddwKwatUqRo4cydKlSxt1HyKNpSbgKKv0Dtq7ggdv71H39a7dNZRXVrNr/2VV\ne8pq2LX3M9VUVAX8/qcd1PTrhnJG7/Zh3YcSgg+mT59OTk7OYRNCU6udEET88tXmUv788TpWby3b\nd8CvDB7Ud+874O+uDv3AbQYZyYlkpCaRkZJIRor33DwtiY4t0rxlqfuW114nMcEwMwxIMMMMLLhN\n8N4nBMvNgo/gcqu9HCPBgNrleOskWN3t1d2X99yjTUZjV/UBlBAaaNWqVZx33nkMGTKExYsX079/\nf2bMmEFGRgYFBQXcfvvtlJaW0rZtW6ZPn857771Hfn4+V199Nenp6XzwwQf8+te/5tVXX6W8vJxT\nTjmFp556KuTTMQfbR6dOnTjjjDMYNmwYb7/9NsXFxTz99NOcdtpplJWVMXbsWJYuXUrv3r1Zv349\nTz75JHPnzqW8vJzc3Fz69+/PAw88QE1NDePHj+f999+nS5cuzJs3j/T09DDXqMSjHeVV/PWTDcwt\nWMviNcUkJhjdstLJSEmiWWoiLdOT6dwyjfSURJqlJHkH72SvbM+yOmX7LUtLTtApzhDEVEK479Vl\nLF+/s1G32a9zC+65sH+966xcuZKnn36a4cOHM27cOCZPnsxtt93GLbfcwrx582jXrh1z5szhrrvu\nYtq0aTzxxBM8+uij5OXlATBx4kTuvvtuAK655hpee+01LrzwwsPGVlVVdch9AFRXV/PRRx8xf/58\n7rvvPt566y0mT55MVlYWy5cvZ+nSpeTm5gLw0EMP8cQTT1BYWAh4ie6LL75g9uzZTJ06lcsuu4yX\nXnqJMWPGHHVditRWE3C89+UW5hYU8cayb9hdHaBX+0zuPL8Pl+R2oX2LNL9DjDsxlRD80q1bN4YP\nHw7AmDFjmDRpEueddx5Lly7l3HPPBaCmpoZOnTod9PNvv/02jzzyCGVlZWzbto3+/fuHlBBWrlxZ\n7z5GjRoFwJAhQ1i1ahUACxYs4LbbbgMgJyeHAQMGHHL7PXv23Jswam9DpCG+2lzKSwVFvLx4Hd/s\nrKBlejKXD+3G6CFdOaFLS/2S91FMJYTD/ZIPl/2/wGaGc47+/fvzwQcf1PvZiooKbrrpJvLz8+nW\nrRv33ntvyEM5D7eP1FRvREJiYiLV1dUhbfNgn9+zjfLy8iPehggc/JTQt45vx90X9uPsvu1JTUr0\nO0QhBq5DiARr1qzZe1CeNWsWp556Kr1792bz5s17l1dVVbFs2TIAmjdvTklJCcDeg3/btm0pLS1l\n7ty5Ie+3vn0cyvDhw3nhhRcAWL58OZ9++unesuTkZKqqqkLev0h9agKOdz7fzK2zP+bEB97izlc+\npaSimjvP78MHPz2LaWOHcv4JnZQMIkhMtRD80rt3b5588knGjRtHv379+MEPfkBKSgpz587l1ltv\nZceOHVRXV/PDH/6Q/v37M3bsWG688ca9ncrjx48nJyeHjh07MnTo0JD3W98+DuWmm27i2muvpV+/\nfvTp04f+/fvTsmVLACZMmMCAAQMYPHgwDzzwQIPrReKTTglFL3PO+R1DyPLy8tz+90NYsWIFffv2\n9Smi6BuvX1NTQ1VVFWlpaXz11Vecc845rFy5kpSUlKPant/1L5HhUKeERg/pqlNCEcDMCpxzeYdb\nTy2EOFNWVsaZZ55JVVUVzjkmT5581MlA4ltNwLGg1iihSo0SinpKCA2UnZ0dNa0D8PovdNc5aYiv\nNpcyt6CIV2qdErpCp4RighKCiByWRgnFByUEETkonRKKP0oIIlLHl5tKeWmxTgnFIyUEEdnrZ3/+\nlOcWrtEpoTilC9N8MH36dNavX3/Y9caOHXtEF6rt8Yc//IEZM2YcsHzVqlXk5OQAUFhYyPz58/eW\n3XvvvTz66KNHvC+JHXMLinhu4RquHtZdF47FKbUQfBDu6a9vvPHGw65TWFhIfn4+559/flhikOjy\n5aZSfv7npQzr2Zr7L84hMUGnheKRWggNtGrVKvr06cPVV19N3759GT16NGVlZYA3NfW3vvUthgwZ\nwogRI9iwYQNz587dO/11bm4u5eXl3H///QwdOpScnBwmTJhAfRcLbtq0iSFDhgCwZMkSzIw1a9YA\ncOyxx1JWVlbn135BQQEDBw5k4MCBPPnkkwBUVlZy9913M2fOHHJzc5kzZw7gTWVxxhlncMwxxzBp\n0qSw1ZlEloqqGibOWkx6SiK/v2KQkkEci72E8MwFBz7em3T05SFYuXIlN910EytWrKBFixZMnjx5\n79TUc+fOpaCggHHjxnHXXXcxevRo8vLymDlzJoWFhaSnpzNx4kQWLVrE0qVLKS8v57XXXjvkvtq3\nb09FRQU7d+7k3XffJS8vj3fffZfVq1fTvn17MjLq3kTjuuuu4/HHH2fJkiV7l6WkpHD//fdz+eWX\nU1hYyOWXXw7AZ599xhtvvMFHH33Efffdp3mN4sQvXlvOZ9+U8NhlA+nYUiOH4plvCcHMupnZ22a2\n3MyWmdltfsXSUPtPf71gwYI6U1Pn5ubyy1/+kqKiooN+/u2332bYsGGccMIJ/POf/zzsBHWnnHIK\n7733Hu+88w533nkn77zzDu+++y6nnXZanfWKi4spLi7m9NNPB7x7LdTnggsuIDU1lbZt29K+fXs2\nbtwYahVIlPrrJxuY+eEabjj9GM4M8+0ZJfL52YdQDfy3c26xmTUHCszs78655Q3a6nV/DW/5QTT1\n9Nenn3763lbBxRdfzMMPP4yZccEFobVoDmX/6a6PZspsiR5rtpbx05c+IbdbK348orff4UgE8K2F\n4Jzb4JxbHHxdAqwAuvgVT0M09fTXp512Gs899xy9evUiISGB1q1bM3/+fE499dQ667Vq1YpWrVqx\nYMECAGbOnLm3rHYMEn8qqwPcMnsxGDx+5SCSE2Pv7LEcuYj4FphZNjAI+NDfSI7Onumv+/bty/bt\n2+tMf33HHXcwcOBAcnNzef/99wH2Tn+dm5tLamrq3umvR4wYEdL019nZ2Tjn9p4KOvXUU2nVqhVZ\nWVkHrPvMM89w8803k5ubW6ez+swzz2T58uV1OpUlfjzyt89YUrSDR747gG6tw3/zdokOvk9/bWaZ\nwL+BB5xzLx+kfAIwAaB79+5DVq9eXafc7+mXo23668bmd/3LkfvnZxsZNz2fa07qwS8uyfE7HGkC\noU5/7WsLwcySgZeAmQdLBgDOuSnOuTznXF67du2aNkCRGLNhRzn//cIS+nZqwV0XKJFLXX6OMjLg\naWCFc+43fsXRUNE2/bXEr+qaALfNLmR3dYAnrhpEWrKuQJa6/GwhDAeuAc4ys8LgQ5fNioTJpH98\nwUertvHLS3I4tl2m3+FIBPJt2KlzbgHQKJdEOuc0A6MP/O5/ktC9/+UWHn/7S747uCujBnf1OxyJ\nUBExyqgh0tLS2Lp1qw5OTcw5x9atW0lL05WtkW5zyW5um1PIMW2bcf/F/f0ORyJY1E9u17VrV4qK\niti8ebPfocSdtLQ0unbVr81IFgg4bn+hkB3lVcwYdyLNUqP+T17CKOq/HcnJyfTs2dPvMEQi0lPv\nfM27X2zhgUtz6Nuphd/hSISL+lNGInJwBau38eibK7nghE5cdWJ3v8ORKKCEIBKDissquXV2IZ1b\npfGr756gQRcSkqg/ZSQidTnn+MncT9hUUsHcG0+hRVqy3yFJlFALQSTGPPv+Kt5cvpE7zuvDwG6t\n/A5HoogSgkgMWbpuBw/O/4yz+rTn+lM12EKOjBKCSIwo3V3NxFmLad0shUe/N1D9BnLE1IcgEgOc\nc9z1yqes2VbG8xNOpnWzFL9DkiikFoJIDHgxv4h5hev50TnHc2LP1n6HI1FKCUEkyn2xsYS7/7KU\nU45tw01nHud3OBLFlBBEolh5ZQ03z1pMs5Qkfnd5LokJ6jeQo6c+BJEodv9ry/h8Yykzxp1I+xaa\naFAaRi0EkSj1lyXrmf3RWn5wxrGcfrzuJigNp4QgEoVWbdnFnS9/yuDurbj93OP9DkdihBKCSJTZ\nXV3DLbM/JjHBmHTlIJIT9WcsjUN9CCJR5qHXP+PTdTt46pohdM3K8DsciSH6aSESRf6+fCPPvLeK\nsadkM6J/R7/DkRijhCASJdYVl/PjF5eQ06UF/3t+H7/DkRikhCASBaprAtw2+2OqawI8fuVgUpMS\n/Q5JYpD6EESiwG/f+pz81dv5/RW59GzbzO9wJEaphSAS4d79YjOT//UVl+d14+LcLn6HIzFMCUEk\ngm0qqeBHcwo5rl0m917U3+9wJMbplJFIhAoEHLfPWULp7mpmjT+J9BT1G0h4KSGIRKj/+/dXLPhy\nCw+NOoHjOzT3OxyJAzplJBKBFq3axmNvruTCgZ25fGg3v8OROHHYhGBmB9yY9WDLRKRxbN9Vya2z\nP6Zb6wwevDRHt8KUJhNKC+Glgyyb29iBiIh3K8z/mbuELaW7eeLKwTRPS/Y7JIkjh+xDMLM+QH+g\npZmNqlXUAtDE6yJhMO29Vby1YhP3XNiPE7q29DsciTP1dSr3BkYCrYALay0vAcaHMyiRePRJUTEP\nvb6Cc/t1YOwp2X6HI3HokAnBOTcPmGdmJzvnPgjHzs1sGl7S2eScywnHPkSiwc6KKibO+ph2man8\nevQA9RuIL0IZdvqlmd0JZNde3zk3rhH2Px14ApjRCNsSiUplldXcMKOAdcXlzJlwEq0yUvwOSeJU\nKAlhHvAu8BZQ05g7d869Y2bZjblNkWhSuruacc8sIn/1Nn57eS552a39DkniWCgJIcM5d0fYIxGJ\nMyUVVYx9ZhGFa4uZdOUgRg7o7HdIEudCGXb6mpmdH/ZIDsHMJphZvpnlb9682a8wRBrVjvIqxjz9\nEUvWFvOEkoFEiFASwm14SaHCzHaaWYmZ7Qx3YHs456Y45/Kcc3nt2rVrqt2KhE1xWSVj/vghy9fv\nYPLVg/nOCZ38DkkECOGUkXNOk6iINJJtu7xk8OWmUp66Zghn9engd0gie4UydYWZ2Rgz+3nwfTcz\nO7Exdm5ms4EPgN5mVmRm1zfGdkUi0ZbS3Vw1dSFfbS5l6rV5SgYScULpVJ4MBICzgF8ApcCTwNCG\n7tw5d2VDtyESDTaVVHD11A9Zu72MaWOHMvy4tn6HJHKAUBLCMOfcYDP7GMA5t93MNFBaJEQbd1Zw\n5dSFfLOjgmfGnsjJx7bxOySRgwolIVSZWSLgAMysHV6LQUQOY8OOcq6a+iGbdlbw7LgTGarrDCSC\nhTLKaBLwCtDezB4AFgAPhjUqkRhQtL2My59ayJaS3cy4fpiSgUS8UEYZzTSzAuBswIBLnHMrwh6Z\nSBRbu62MK6YspKSiij99fxi53Vr5HZLIYdU3/XUL59xOM2sNbAJm1ypr7Zzb1hQBikSbVVt2cdXU\nheyqrGHW+JPI6aJprCU61NdCmIU3E2kBwf6DIAu+PyaMcYlEpa83l3Ll1IVUVgeYPf4k+nVu4XdI\nIiGrb/rrkcFn3S5TJARfbirhyqkfEgg4Zk84iT4dlQwkuoRyYdqlZtay1vtWZnZJeMMSiS4rvynh\niikLcQ6eVzKQKBXKKKN7nHM79rxxzhUD94QvJJHosnz9Tq6cupDEBGPODSfRq4Nme5HoFMp1CAdL\nGqF8TiTmLV23gzFPf0h6ciKzx59EdttmfockctRCaSHkm9lvzOzY4OM3eB3NInFtydpirpq6kGYp\nScyZcLKSgUS9UBLCLUAlMCf42A3cHM6gRCLd4jXbGfPHD2mZkcycG06ie5sMv0MSabBQLkzbBfy0\nCWIRiQr5q7Yx9plFtM1MYdb4k+jcKt3vkEQaRX0Xpv3OOfdDM3uVutchAOCcuyiskYlEoIVfb2Xc\n9EV0bJHGrPEn0bFlmt8hiTSa+loIM4LPjzZFICKR7v0vtzDu2UV0zcpg1vhhtG+uZCCxpb6E8Gu8\n+YvOd87d0UTxiESkdz7fzPgZ+WS3acbM8cNom5nqd0gija6+hNDJzE4BLjKz5/GmrNjLObc4rJGJ\nRIi3P9vEDc8VcGy7TGZ+fxitm+l2IBKb6ksIdwM/B7oCj1E3ITi8O6iJxLS3lm/kppmLOb5jJs9d\nP4xWGUoGErvqSwgbnHPfMbO7nXP3N1lEIhHib0u/YeKsxfTv3IIZ44bRMiPZ75BEwqq+6xAmBZ81\nb5HEnb9+soGbZy3mhK4t+dP3lQwkPtTXQqgysylAFzObtH+hc+7W8IUl4p95heu4/YUlDOrWimeu\nG0rzNCUDiQ/1JYSRwDnACDRVhcSJlxcX8eMXlzA0uzXTxg6lWaqm7ZL4Ud/9ELYAz5vZCufckiaM\nScQXLyxayx0vf8LJx7Thj9fmkZGiZCDxJZS5jMrN7B9mthTAzAaY2c/CHJdIk5r14Rp+8tInnHpc\nW6aNHapkIHEplIQwFfhfoArAOfcJcEU4gxJpSjM+WMWdr3zKmb3bMfW/8khLTvQ7JBFfhPIzKMM5\n95FZnevSqsMUj0hY1QQcX2wqIX/Vdhav3k7Bmu2s3lrGOX078OTVg0hNUjKQ+BVKQthiZscSnODO\nzEYDG8IalUgjKamoonBtMQWrt1OwejuFa4op2e39nmmbmcLg7llce3I2Y07qQUpSKA1mkdgVSkK4\nGZgC9DGzdcB/gKvDGpXIUXDOsWZb2d6Df8Hq7azcWIJzYAa9OzTnotzODOmRxZAeWXRvncF+LV+R\nuBbK/RC+Bs4xs2ZAgnOuJPxhiRxeRVUNy9bvqJUAitlSuhuAzNQkBnVvxYj+HcnLzmJgt1a00PUE\nIvU6bEIws5bAPcDpwff/Bu53zu0Ic2widWwqqfDO+wcfS9ftpLImAECPNhmc3qstg4O//o/v0JzE\nBP36FzkSoZwymgYsBS4Lvr8GeAYYFa6gRGoCjpXflFCwxuv8zV+9jbXbygFISUzghK4tuW54NoN7\nZDG4exbtmms6apGGCiUhHOuc+26t9/eZWWFj7NzMzgN+DyQCf3TOPdQY25Xos7Oiio/XeJ2/i1dv\n5+M129lVWQNA28xU8npk8V8neQkgp0sLjQYSCYNQEkK5mZ3qnFsAYGbDgfKG7tjMEoEngXOBImCR\nmf3FObe8oduWyFFVE2DX7mpKd1eza3dN8Nl7FJdX8UnRDhav3s7nm7zO3wSD3h1bcOngLuT1aM2Q\nHll0zUpX569IEwglIfwAeDbYlwCwHRjbCPs+Efgy2GlN8CY8FwNKCD4KBBxlVTW1DuLVlFYEX1dW\nU7q7Zu8Bfd/Bfd+Bfs96e5ZVVgfqbN8IkESARGowHEmpmQzqkcX5J3RiSI8sBnZrqcnkRHwSyiij\nQmCgmbUIvt/ZSPvuAqyt9b4IGNZI267jo/9sY3fhi6RU7QvdARWpbVjb4WwAemz4GylVdfvJK1La\nsCZYnr3hdVKq6g6wKj+gfMd+5W1ZHSzvueF1Uqt2eBdzOAfOUZrSlq/bn0PAOY7f8BqpVcXgHM45\nHI7S5HasaDuCgHPkbJxHenVx8LMBcLAjpR2Frc8n4Bx5m1+hWdU2nHPBf6BjW3J7Pmw1Eofj1K1z\nyazevnffNYEavqENLyddwK7d1VxV9ida12zBBar3HrD/4zrxWLXXdfRw0hS62SY6W4Akakiihq+s\nBw+nTqRZahKPVtxHl8B6kgiQZDUkWg3ftOnH/AGTaJaaxHc/HE3Lkq8w3N762d1hMEk3/NPr/H3l\nRigogc87QmZHaN4B2vSCHifv/fegVoJIWIUyyuhB4BHnXHHwfRbw3865JpnPyMwmABMAunfvflTb\neHXJeq75+Pccn7CuzvL8wPH8vLIDAG+mHLz87sqOwfLHD1p+z2HK76vsFCx/4oDyRYHjeaiyKwB/\nT5lCr4OU37rsOBIMXk96luOsbvlienPP6oGYwdjA8xxDUZ3yJdaH3287GTOYWDGP7q4IMBxGgAQ+\nT+nHO50upUebDM5c9xVtq9bjLBESkrCERAZmtafvKXk0S02iz/svklZWTmJSEomJySQkJTOwfV9G\nffscb2dv/gNKN0FCEiR42ziu9THcekovrzzxeijfXqc8tfUx3jkigOrdsPVLWLUAKoq9ZceeBde8\n4r1+Ig8qy6B5R++R2QG6nQi5V3nlmz6DtJbQrB0kah4ikaNhe39RHmoFs4+dc4P2W7bYOTe4QTs2\nOxm41zk3Ivj+fwGcc7861Gfy8vJcfn7+Ee9rZ0UVu3dsxFzd0xckJOPSW3vxlG0FV133R2iw3PaW\n19SOP1ie5b0v34YFaqgjMRnSW+0tJ1CDWQKWQPA5mYT0lphBQmUJhiPBjIQEwywBLBFSMrxtVVXg\ntWss+EvZwBL2HfwCNfvKov2XdFUFlG706rv1Md6yfz0Mxauh5BuvrGQD9PwWfO8Zr/zhbC/hWIKX\nFDI7QJ8L4IyfeuWFsyC1BTTv5LU+Mjt4/z8iccDMCpxzeYdbL5SfUolmluqc2x3ccDrQGGP8FgG9\nzKwnsA5vwryrGmG7B2iRlgxpXetfqXnn+sszD1PerGP95RmHKU9udZjytPrLE2Jo1E1yGmT1qLvs\njDsOXK/2j5mLn/SSRMlGKP3GSxwWrJNAAP5yCwT2m4JryFi48Pfe65oqJQiJe6EkhJnAP8ws+FOM\n64BnG7pj51y1mU0E3sAbdjrNObesoduVOFK7JdTngvrX+9GyAxNG+75e+df/gld/CFfNgXa9wxqy\nSCQLpVP5YTNbgnf3NIBfOOfeaIydO+fmA/MbY1sih2S2r+/hYDLaQuUumHYejJkLXYY0bXwiESKk\n6R2dc39zzv04+GiUZCASMTrmwPVvQGpzePYir8UgEoc0368IeJ3X496AVt1h5vdgnW4jLvFH4/NE\n9mjRCcb+FRZOho4D/Y5GpMkdtoVgZreFskwkJmS0hrN+5g3nLfkG8qf5HZFIkwnllNG1B1k2tpHj\nEIk8Hz4Fr/0I3rq37hBXkRh1yFNGZnYl3nUBPc3sL7WKmgPbwh2YiO/O+pl3sduC30LZNhj529i6\n3kNkP/X1IbyPd+/ktsBjtZaXAJ+EMyiRiJCQ6CWBjNbw7mNQsQNGTYEk3XtBYtMhE4JzbjWwGji5\n6cIRiTBmcPbdkN4aPpoC5cXe1BciMSiUTuVRZvaFme0ws51mVmJmjTXjqUh0OGUi3PSBlwxqqr3E\nIBJjQulUfgS4yDnX0jnXwjnX3DnXItyBiUSclGbe8+s/gWkjYOd6f+MRaWShJISNzrkVYY9EJFr0\nvwR2FMHTI2DrV35HI9JoDpkQgqeKRgH5ZjbHzK7csyy4XCQ+9Twdrn0VqnZ5LYUNGmMhsaG+FsKF\nwUcLoAz4dq1lI8MfmkgE6zIYrvsbJKbAny6F3aV+RyTSYPWNMrquKQMRiTrtjvfmP9q0HFIz/Y5G\npMFCuYXmpIMs3gHkO+fmNX5IIlGkVTfvAfDpXO9+1wMu8zcmkaMUSqdyGpALfBF8DAC6Ateb2e/C\nGJtI9HAOCmfCy+Phwyl+RyNyVEKZ7XQAMNw574bCZvZ/wLvAqcCnYYxNJHqYwRWzYe44eP1/vCkv\nvvWT6L+/tcSVUFoIWUDtE6TNgNbBBLE7LFGJRKPkNLhsBuReDf96EF6/w7ufs0iUCKWF8AhQaGb/\nAgw4HXjQzJoBb4UxNpHok5gEFz0B6VlQVaYWgkQVcyFM62tmnYATg28XOed8uUQzLy/P5efn+7Fr\nkSPjnPdISIDtqyGzPSSn+x2VxCkzK3DO5R1uvfouTOsTfB4MdALWBh8dg8tE5FDMvGRQVQHPjoQ/\njfJmSxWJYPWdMrodmEDdqa/3cMBZYYlIJJYkp8E598LLE2D6BTDmZa+1IBKB6rswbULw+cymC0ck\nBuV8F1JbwgvXeFNdXPNnyOrhd1QiBwhl+usMM/uZmU0Jvu9lZpq6QuRI9DrHSwRlW2H+//gdjchB\nhTLK6BmgADgl+H4d8CLwWriCEolJ3Yd58x81a+t3JCIHFcp1CMc65x4BqgCcc2V4w09F5Eh16Of1\nIdRUwYvXwVdv+x2RyF6hJIRKM0vH60jGzI5FF6SJNEzFTti8EmZdBss1JZhEhlASwr3A34BuZjYT\n+Afwk3AGJRLzmrWB6/4KnXLhxbFQ8KzfEYkcvg/BOfemmRUAJ+GdKrrNObcl7JGJxLr0LPivP8Oc\na+DVW73rFIbf6ndUEsdCmf76OeDfwLvOuc/CH5JIHElpBlc+D3+7Azqe4Hc0EudCGWX0NHAa8Hiw\n/+Bj4B3n3O/DGplIvEhKgZG/3fe+ptqbE0mkiR22D8E59zbwAPBzYCqQB/wgzHGJxKd3fwMzLvaS\ngkgTC+XCtH8A7wGXAyuBoc65Pg3ZqZl9z8yWmVnAzA474ZJI3GjZFVYvgH8/7HckEodCGWX0CVAJ\n5ODdLCdIrYbEAAALTElEQVQnOAy1IZYCo4B3Grgdkdgy4DLvfgrv/Br+oz8PaVqhnDL6kXPudLwD\n+Fa8K5eLG7JT59wK59zKhmxDJGZ95xFoc5w3Id4uDeiTphPKKaOJZjYHrzP5YmAa8J1wByYSt1Iz\nYfQ0KC/WlczSpEIZypAG/AYocM6F3NNlZm8BHQ9SdJdzLuRLM81sAt403HTv3j3Uj4lEt04D4LYl\n0LyD35FIHAnlwrRHj2bDzrlzjuZzB9nOFGAKeHdMa4xtikSFPcng639BWkvoPMjXcCT2abCzSCSr\nroR5t0BCItzwDqS18DsiiWGhjDJqdGZ2qZkVAScDfzWzN/yIQyTiJaXAqClQvBr+ert3n2aRMPEl\nITjnXnHOdXXOpTrnOjjnRvgRh0hU6HEynHEnfPoiFM7yOxqJYb4kBBE5QqfdDtmnwfwfw7av/Y5G\nYpT6EESiQUIijJoKS2ZDK92PWcJDLQSRaNGik9dSSEiE3aV+RyMxSAlBJNps+gwmDYIVuq25NC4l\nBJFo0/oYaNEZ5t0MxWv9jkZiiBKCSLRJSvGmtghUw0vf11TZ0miUEESiUZtjYeTvYO1C+PdDfkcj\nMUKjjESi1YDvedNarP0IAjVeZ7NIAyghiESzCx6DxGQlA2kUOmUkEs2S07xkUPINvP0gBAJ+RyRR\nTAlBJBasfN277ebCyX5HIlFMCUEkFgwZC31Gwlv3wrrFfkcjUUoJQSQWmMHFT0DzjjB3HFTs9Dsi\niUJKCCKxIj0LvvtHKF7jtRREjpBGGYnEku4nefdPyD7N70gkCikhiMSaE0Z7z4EaKNsKme39jUei\nhk4ZicSql66HP10KVeV+RyJRQglBJFblXg0bl8KbP/M7EokSSggisarXuXDyRFj0R1j+F7+jkSig\nhCASy86+BzoPgr9M9EYfidRDCUEklu2ZKrtZO296C5F6aJSRSKxrfQzc/JEmwJPDUgtBJB4kJHo3\n0nn7QW/KbJGDUEIQiRc1lbDsz/DyBCjd7Hc0EoGUEETiRUoGfO8ZKC+GP9+oqbLlAEoIIvGkQ384\n70H48i1Y+KTf0UiEUUIQiTd510PfC+Efv9DII6lDo4xE4o0ZXPQ4bFzmTZctEqQWgkg8Ss+C7FO9\n1xuXgXP+xiMRQQlBJJ6tWgD/Nxw+/pPfkUgEUEIQiWfdT4aep8P8n8Cmz/yORnzmS0Iws1+b2Wdm\n9omZvWJmrfyIQyTuJSR6N9RJaQZzr9NU2XHOrxbC34Ec59wA4HPgf32KQ0Sad4RLn4JNy+GNO/2O\nRnzkS0Jwzr3pnKsOvl0IdPUjDhEJ6nUODP8hZHZQB3Mci4Rhp+OAOX4HIRL3zr1v3+sPJoMLwIDL\nIbOdfzFJkwpbQjCzt4CDDXK+yzk3L7jOXUA1MLOe7UwAJgB07949DJGKyAH+82/4/G/w1j3Q+zsw\n6Bo49mxIjITfkBIu5nxqHprZWOAG4GznXFkon8nLy3P5+flhjUtEgjatgI+fgyXPQ9kW75acl0z2\nOyo5CmZW4JzLO9x6vqR7MzsP+AnwrVCTgYg0sfZ9YcQD3l3Xvnhz31XNW7+CeTfDoDHQ7xJIzfQ3\nTmk0fo0yegJoDvzdzArN7A8+xSEih5OUAn1HQtfgD8zSjbBri5cUHj3ee17zoTqjY4AvLQTn3HF+\n7FdEGkGPU2DiIlj7kXeF87JXoHAW/GgZtOgMgRrdnS1KqYdIRI6cGXQf5j3OewjWLvSSAcDsK73y\nQWOg1wivhSFRQQlBRBomNROOO8d77Rx0zIGPZ3qjlDLawsArYPB/Qbve/sYph6W5jESk8ZjB2Xd7\np4+uegF6nAwf/gE+ecErr6mGih3+xiiHpBaCiDS+xCQ4foT3KN0MFvzt+eVb8OJY6Hexd0qpx3BI\n0O/SSKGEICLhVftK59Y9IfdK+HQufPI8ZGVD7hg46QcavhoBlJpFpOm06w0jfwv/vRJGTYVW3eGj\nKZAY7HjevBKqd/sbYxxTC0FEml5KBgy4zHtU7PRGIgUC8NxoqCzx5lDqNsybljurJ7Q73vvczvXe\nsuRmmkYjDFSjIuKvtBb7Xl/4O2+6jPxpXmc0wNDvwwWPQU0V/KbvvnWT0iA5A/Ku8zqyAzXwp0sh\nJdNLGnsePYZDn/O9EVCfzKlVlul9PrODd1prz4V1Zk33b48wSggiEhkSEuC4s71HxQ7YuQEqd0F6\nrftnXfh7b1nlLqgs9Z7b9/PKaiq9001l2/aVVe7yyvqcD1Vl8MoNB+532A/gOw9BdQU80Gm/hJIB\nA6+Ck2/ytjXjYsCCSSP4nPNdOHE8VJbBrMtqlSV4r/te5CWtqnJ44doDy3ufD4Ou9spfufHA8l7f\n9lpSTUAJQUQiT1pL71FbYjIMGXvozySnw/VvHLh8zy//pHS49eMDE0qrHvvWPf1/6pZV7vISwx4p\nmYALbjP4vLdF4SBQXbfMBbxEA97r0m9qlQeXlW31ygM13k2K9v98uz6h1Vkj8G2206Oh2U5FRI5c\nqLOdapSRiIgASggiIhKkhCAiIoASgoiIBCkhiIgIoIQgIiJBSggiIgIoIYiISFBUXZhmZpuB1UBL\n4GB32dh/+aHW81NbYEsI64Uae6h1cSTbbEp+1Uc81EV968ZjfURzXUDD6qOHc67dwVauwzkXdQ9g\nSijLD7Wez7HnN+TfeLR1ofqIv++G6iN26iJc34/9H9F6yujVEJcfar1oEGrsodbFkWwzEjV2fcRD\nXdS3bjzWRzzUBTQg/qg6ZRQLzCzfhTCnSLxQfeyjuqhL9VFXU9RHtLYQotkUvwOIMKqPfVQXdak+\n6gp7faiFICIigFoIIiISpIQgIiKAEoKIiAQpIUQYM2tmZvlmNtLvWPxkZpeY2VQzm2Nm3/Y7Hj8E\nvwvPBuvhar/j8Zu+E3WF41ihhNBIzGyamW0ys6X7LT/PzFaa2Zdm9tMQNnUH8EJ4omwajVEXzrk/\nO+fGAzcCl4cz3qZ0hHUzCpgbrIeLmjzYJnAk9RGr34k9juLvptGPFUoIjWc6cF7tBWaWCDwJfAfo\nB1xpZv3M7AQze22/R3szOxdYDmxq6uAb2XQaWBe1Pvqz4OdixXRCrBugK7A2uFpNE8bYlKYTen3s\nEWvfiT2mE/rfTViOFUmNubF45px7x8yy91t8IvClc+5rADN7HrjYOfcr4IBmnpmdATTD+48vN7P5\nzrlAOOMOh0aqCwMeAl53zi0Ob8RN50jqBijCSwqFxOiPtyOpDzNbQQx+J/Y4wu9GJmE4VighhFcX\n9v3CA+8PfNihVnbO3QVgZmOBLdGYDOpxRHUB3AKcA7Q0s+Occ38IZ3A+O1TdTAKeMLMLiP7pFI7E\noeojnr4Texy0LpxzE6HxjxVKCBHIOTfd7xj85pybhHdAjFvOuV3AdX7HESn0nThQYx8rYrIZGkHW\nAd1qve8aXBaPVBeHprqpS/WxT5PWhRJCeC0CeplZTzNLAa4A/uJzTH5RXRya6qYu1cc+TVoXSgiN\nxMxmAx8Avc2syMyud85VAxOBN4AVwAvOuWV+xtkUVBeHprqpS/WxTyTUhSa3ExERQC0EEREJUkIQ\nERFACUFERIKUEEREBFBCEBGRICUEEREBlBBEGsTMOprZ82b2lZkVmNl8Mzve77hEjobmMhI5SsEZ\nWV8BnnXOXRFcNhDoAHzuZ2wiR0MJQeTonQlU1Z510zm3xMd4RBpEp4xEjl4OUOB3ECKNRQlBREQA\nJQSRhlgGDPE7CJHGooQgcvT+CaSa2YQ9C8xsgJmd5mNMIkdNCUHkKDlvquBLgXOCw06XAb8CvvE3\nMpGjo+mvRUQEUAtBRESClBBERARQQhARkSAlBBERAZQQREQkSAlBREQAJQQREQlSQhAREQD+H4I+\nyQPMwNctAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x113763e10>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(params, weights[:, 0],\n",
    "        label='petal length')\n",
    "plt.plot(params, weights[:, 1],\n",
    "        linestyle='--', label='petal width')\n",
    "plt.ylabel('weight coefficient')\n",
    "plt.xlabel('C')\n",
    "plt.legend(loc='upper left')\n",
    "plt.xscale('log')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "\n",
    "\n",
    "我们可以看到随着C的减小，权重系数也减小。\n",
    "\n",
    "\n",
    "**Note** 本章只是简要介绍了逻辑斯蒂回归模型，如果你还意犹未尽，推荐阅读 Logistic Regression: From Introductory to Advanced Concepts and Applications, Sage Publications.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## 支持向量机\n",
    "\n",
    "另一个经常使用的机器学习算法是支持向量机(support vector machine, SVM)，SVM可以看做是感知机的扩展。在感知机算法中，我们最小化错误分类误差。在SVM中，我们的优化目标是最大化间隔(margin)。间隔定义为两个分隔超平面(决策界)的距离，那些最靠近超平面的训练样本也被称为支持向量(suppor vectors)。可以看下图：\n",
    "\n",
    "\n",
    "![](https://ooo.0o0.ooo/2016/06/20/5767ae1337091.png)\n",
    "\n",
    "\n",
    "## 最大化间隔\n",
    "\n",
    "最大化决策界的间隔，这么做的原因是间隔大的决策界趋向于含有更小的泛化误差，而间隔小的决策界更容易过拟合。为了更好地理解间隔最大化，我们先认识一下那些和决策界平行的正超平面和负超平面，他们可以表示为：\n",
    "![](https://ooo.0o0.ooo/2016/06/20/5767b4c0a36f8.png)\n",
    "用(1)减去(2)，得到：\n",
    "![](https://ooo.0o0.ooo/2016/06/20/5767b505b8b47.png)\n",
    "对上式进行归一化，\n",
    "![](https://ooo.0o0.ooo/2016/06/20/5767b57928a93.png)\n",
    "其中，![](https://ooo.0o0.ooo/2016/06/20/5767b59ada318.png)。\n",
    "\n",
    "上式等号左边可以解释为正超平面和负超平面之间的距离，也就是所谓的间隔。\n",
    "\n",
    "现在SVM的目标函数变成了最大化间隔$\\frac{2}{||w||}$,限制条件是样本被正确分类，可以写成：\n",
    "\n",
    "![](https://ooo.0o0.ooo/2016/06/20/5767b6164a8ac.png)\n",
    "\n",
    "\n",
    "上面两个限制条件说的是所有负样本要落在负超平面那一侧，所有正样本要落在正超平面那侧。我们用更简洁的写法代替：\n",
    "\n",
    "\n",
    "![](https://ooo.0o0.ooo/2016/06/20/5767b6675272a.png)\n",
    "\n",
    "实际上，使用二次规划(quadratic programming)最小化$\\frac{1}{2}||w||^{2}$很容易，但是二次规划显然超出了本书的内容，如果你对SVM感兴趣，推荐阅读Vladimir Vapnik写的 The Nature of Statistical Learning Theory, Springer Science&Business Media或Chris J.C. Burges写很棒的解释A Tutorial on Support Vector Machines for Pattern Recognition.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 使用松弛变量解决非线性可分的情况\n",
    "\n",
    "虽然我们不想深挖SVM背后的数学概念，但还是有必要简短介绍一下松弛变量(slack variable) $\\xi$,它是由Vladimir Vapnik在1995年引入的，借此提出了软间隔分类(soft-margin)。引入松弛变量的动机是原来的线性限制条件在面对非线性可分数据时需要松弛，这样才能保证算法收敛。\n",
    "\n",
    "\n",
    "松弛变量值为正，添加到线性限制条件即可:\n",
    "\n",
    "![](https://ooo.0o0.ooo/2016/06/20/5767d4c235d44.png)\n",
    "\n",
    "新的目标函数变成了:\n",
    "\n",
    "\n",
    "![](https://ooo.0o0.ooo/2016/06/20/5767d54416998.png)\n",
    "\n",
    "使用变量C，我们可以控制错分类的惩罚量。和逻辑斯蒂回归不同，这里C越大，对于错分类的惩罚越大。可以通过C控制间隔的宽度，在bias-variance之间找到某种平衡：\n",
    "\n",
    "\n",
    "![](https://ooo.0o0.ooo/2016/06/20/5767d62aeaa9b.png)\n",
    "\n",
    "\n",
    "\n",
    "这个概念和正则化相关，如果增大C的值会增加bias而减小模型的方差。\n",
    "\n",
    "\n",
    "\n",
    "我们已经学会了线性SVM的基本概念，下面使用sklearn训练一个模型："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.svm import SVC"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "svm = SVC(kernel='linear', C=1., random_state=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SVC(C=1.0, cache_size=200, class_weight=None, coef0=0.0,\n",
       "  decision_function_shape=None, degree=3, gamma='auto', kernel='linear',\n",
       "  max_iter=-1, probability=False, random_state=0, shrinking=True,\n",
       "  tol=0.001, verbose=False)"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "svm.fit(X_train_std, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAEKCAYAAAASByJ7AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xm4FPWd7/H3Bw4IiKgI4oJKogmKjoISN9SomRh3H42O\n45YYnTE+MRHikmhyb0LM5M5EJ7luUWI2o9EY1xtjdFyiZsEVEBVFonEBN0ARARHknPO9f3Q1NHBO\nn+Kcrq5ePq/n6ae7uqurvn3E/vTv96v6lSICMzOzXnkXYGZmtcGBYGZmgAPBzMwSDgQzMwMcCGZm\nlnAgmJkZ4EAwM7OEA8HMzAAHgpmZJVryLmBdSEMCRlR0m1uNnlfR7ZmZ1Zo50+e8ExFDu1qvrgKh\nEAZTKrrFOdML95cuuKyi2zUzqxUTBk94Lc167jJKTBg8ngmDx+ddhplZbhwIa3AwmFmzciB0wqFg\nZs2mzsYQ1rbxxiuYOPF1tttuGb0qHW+PH7RqP1stqvDGq2NJnyXMGjyLFb1X5F2KmdW4ug+EiRNf\nZ/fdN6ClZQSg7Hb0IWw1em52289ARLBkwRJYADOGzsi7HDOrcXUfCNtttyz7MEjMmT5s5eN6CAdJ\nDBw8kIHvDMy7FDOrA3U/hlDoJso+DNY0Z/qw1QKiVknV/9uYWX2q+0DIWz2EgplZGg6ECpgzfRi/\nu+opdt92P/bcfk+uuPiKvEsyM1tnDoQKaGtr4+KLz+Kyy+7hxutnccdNdzDr+Vl5l2Vmtk7qflB5\nXezwhU/R8t7acxe1brwpM697stvbfe65J9hqq+0YPvzjAOy/78n87qeT+c5lI7u9TTOzamuqQGh5\nbx6tG689v1NHIbEu5s9/g2HDtlq5PGzYcGbMeHzl+EI9HJFkZuYuoyqolyOSzKy5ORAqYOjQLZk7\nd87K5blzX2fo0C3XWs+hYGa1zIFQAaNGfYrZs1/kjTdeYcWKj7j//pvYb78jO1y32FpwOJhZrWmq\nMYSstLS08I1vXMnZZ3+OtrY2jjzyNLbddscu3+cxBjOrJU0VCK0bb9rpUUY9NW7coYwbd2i33jtn\n+jCHgpnlrqkCoSeHlmat3uZJMrPG4zGEGuTxBTPLQ1O1EOqJWwxmVm1uIdQBH5VkZtXgQKgjDgUz\ny1JuXUaStgKuA4YBAVwTEZflVU+9cFeSmWUlzxZCK3BuRIwC9gTOkjQqx3q67aKLTuOggzbl+ON3\nqup+3WIws0rKLRAi4q2ImJY8XgzMBNae76Hi+y2/3B2HH34ql1/+Pz3fUDf4zGczq5SaGEOQNAIY\nAzzewWtnSJoiaQrM79F+br8dbrhhVQhEFJZvv71Hm2XXXfdj0KDBPdtIBTgYzKwncg8ESQOB24AJ\nEbFozdcj4pqIGBsRY2HtqavTioClS+G++1aFwg03FJaXLq1MS6FWOBTMrDtyPQ9BUh8KYXBDRPTw\nd3pX+4KTTio8vu++wg3goIMKzzfateg9T5KZravcWgiSBPwCmBkRP67OPleFQlEjhkGpOdOH8d6c\nQXmXYWZ1IM8uo3HAKcCBkqYnt+7NDpdSsZuoVOmYQiObMHg8EwaPz7sMM6thuXUZRcTfgKr9Ni8d\nMyh2ExWXoWcthW9/+wSmTn2YhQvf4bDDhnPGGd/jqKNOr1zxFVQaCpcu8GkfZrZK08xlJMGAAauP\nGRS7jwYM6Fm30Q9+8NvKFFllEwaPdyiY2UpNEwgAxxxTaCkUv/yLodDIYwhdcSiYWVFTBQKs/eXf\nzGFQ5G4kM4MaOA/BaosHn82alwPBOuRgMGs+DgQry8Fg1jwcCJaKQ8Gs8TXdoHIW3n57DhMnfoEF\nC+YC4uijz+CEExrvC7QYCh54NmtMDoQKaGlpYcKEH7H99rvywQeL+cIXdmOPPT7Lxz9el5d36JKP\nSjJrTE3XZbSg/6vM2Pw2pmz9K2ZsfhsL+r/a420OGbI522+/KwDrr78BI0bswPz5b/R4u/XAXUlm\njaOpWggL+r/Ky0MfQu296d22Hst7f8DLQx+C+Qcw+MMRFdnHm2++yqxZT7HjjntUZHv1wC0Gs8bQ\naSBIOibF+5dFxN0VrCdTb240tRAG0QeA3tGHtvbC85UIhKVLl/DNb36ec865lIEDm3OGUZ/5bFa/\nyrUQfgb8nvIT0O0H1E0gLOuziN5t6632XK9oYVmfta7Ls85aW1fwzW9+noMPPokDD0yTpY3Lg89m\n9alcINwTEaeVe7Ok31S4nkz1WzGI5b0/WNlCAGhXK/1W9OzXfETw/e+fzogRO3DSSef0tMyG4a4k\ns/rS6aByRJzc1ZvTrFNLtli4G9GrjTatIIjCfa82tli4W4+2+/TTk7n77uuZMuVBTjxxNCeeOJrJ\nk+um4VQVHnw2q33dHkPI+pKXWRj84QiYfwBvbjSVZX0W0W/FILZ4d7cejx+MHr0PTz7ZBFfZ6SF3\nJZnVtnJdRkck95sCewMPJssHAI8AdRcIUAiFSh1RZN3jYDCrTZ0GQkR8CUDSfcCoiHgrWd4cuLYq\n1VlDczCY1ZY0J6ZtVQyDxFxg64zqsSbk8QWz2pDmxLQ/SboXKF4n8njggexKsmbkI5LM8tdlIETE\nVyUdTeGcA4BrIuKObMuyZuauJLN8pJ3LaBrwx4j4OnCvpA0yrMkMcFeSWbV1GQiS/h24Ffhp8tSW\nwP/Lsqh6snjxQm655apuv//GGy9l2bKlPa5j6tSHefrpR3q8nVpTvECPw8Ese2laCGcB44BFABHx\nIoVDUY1CINx6a/cD4aabKhcIzzzTeIFQyqFglq00g8rLI+IjqTClkaQWwGdhJa688gLeeOMfnHji\naPbY47OMH38J119/CffffzMrVixn//2P5stf/h4ffvgBF174L8yb9zptbW2cfvr/ZsGCucyf/yZn\nnnkAG200hEmTHlpt21dccQF//eud9O7dwh57HMSECf/Ne+/N5z//80zefns2AOeeeylDh27JbbdN\nonfv3txzz284//wrGDNm3zz+HJnz4LNZdtIEwp8lfQvoL+mzwFeAP2RbVv346lf/i3/8YwY33jgd\ngMceu4/Zs1/k179+gojg3HOPZNq0v7Bw4XyGDNmCSy/9IwBLlrzPwIEbcuONP2bSpIfYaKMhq213\n4cJ3efjhO7j11heQxOLFCwH40Y/Gc+KJX2f06H14++3ZfO1rn+OWW2by+c+fSf/+AznllPOq+wfI\nkWdWNausNIFwAXA68CzwZeDuiPhZplXVscceu4/HH7+Pk04aA8CHHy5hzpwXGT16Xy699FyuuOKb\n7LPP4V3+gh84cEPWW68f3//+6eyzz+Hsu+/hADzxxAO8/PLzK9f74INFLF26JLsPVON8RJJZ5aQJ\nhJOAm0pDQNLhEXFXdmXVr4jg1FMv5JhjvrzWa9dfP43Jk+/m6qv/F5/61Gf493//TqfbaWlp4dpr\nn+DJJ//En/50K7fcciVXX/0g7e3t/OpXj7Heev2y/Bh1x11JZj2XZlD5CuCvknYoee6ijOqpOwMG\nbMDSpYtXLu+11+e4885frvzVPm/eGyxYMI/589+kX78BHHroyZxyyvnMmjVt5fs/+GDxWttdunQJ\nS5a8z7hxh3LOOf+XF198GoA99zyIm2++YuV6s2ZN77COZubBZ7PuSdNCeIVCl9GtkiZGxC2Uv2hO\nU9loo03YZZdxHH/8Tuy99yGMH38Jr7wyk9NO2wuAAQMGctFFv2HOnJe4/PLzkXrR0tKHCy64GoCj\njz6Ds88+mKFDt1htUHnp0sWce+5RfPTRMiKCCRN+DMB5513OD394FiecsDNtba2MGbMfF144iX33\nPYILLjiWP//59w09qJyWWwxm604R5Q8YkjQtInaVNITC9BVPAwdFxM7VKLDUWCmmrnGA0z33zGTI\nkB06eYcBvPPOTA45pLn/Rg4Fa2YTBk+YGhFju1ovTZfRWwAR8Q7wOQqHnO7Us/K6L9w4sW7wyW1m\nXesyECLisJLH7RFxfkSknfIiE4EcDNYtDgWzzpW7YtqlETFB0h/o4ES0iDgy08pSCMTd7c9TKM8B\n0bGgvT3vGmqLxxfMOlZuUPn65P6/s9q5pF8ChwPzIqJb3VA7vHQ/Gwx+n1db9sChsKagtfVdXnrJ\nh6h2xucxmK3S5aBypjuX9gOWANelCYSxUkxZ47kVG2/M6xMnsmy77Xit18cyqbNetbfDSy/1Y+LE\n4bz3Xp+8y6kLDgZrRGkHlTsNBEnPUmbOokodZSRpBHBXdwOhw216qiXrIQeDNZK0gVCuy+jw5P6s\n5L7YhXQyVZzcTtIZwBmQ/rqdgRwK1iOeJ8maUZrzEJ6KiDFrPDctInatSAEZtBBW276DwSrA4WD1\nrJLnIUjSuJKFvVO+ryb48FSrBJ/HYM0gzdQVpwG/krRhsrwwea5uFEPBrQXrKR+VZI2sbCBI6gVs\nFxG7FAMhIt6v1M4l/RbYHxgi6XXguxHxi0ptf00eW7BKcTBYIyrb9RMR7cA3ksfvVzIMkm2eEBGb\nR0SfiBieZRis3KfPcrYKcjeSNZI0YwEPSDpP0laSBhdvmVeWMYeCVYrHF6xRpBlDOD65P6vkuQA+\nXvlyqstdSFZJ1e5GigCp82WzddVlIEREQ5/+6wFnq7RqBMMj1+7EZiPn87E95yIVwqC9Vah30Ktu\njgG0WpOmhYCknYBRwMpJcSLiuqyKyoNbC1ZpWU2iFwGbjZzPNp+aR3sb9OpdeL5XSxDtbilY93X5\nW0LSdylcRvMK4ADgYiD3mU6z4AFny0olxxgk+Niec4GgvbX3avMGqJfDwLovTePyWOAzwNsR8SVg\nF2DD8m+pbw4Fy0IlB5+lVS2D0n+uDgPriTSB8GFy+GmrpEHAPGCrbMvKn1sLlpVKBEMEtLcVF1Z/\n3qy70gTCFEkbAT8DpgLTgEczraqGOBQsK90Nhgh45bFhgOjV0rZaC6E4hrBmMDgoLI00l9D8SkQs\njIhJwGeBLyZdR03DrQXL0rqGggRvzxrKa09uSq/eq7qJ2lsFgtaPevHKY8NWhkAELHx9IMsW+5oY\nVl656yGUnc00IqZlUlEZ3ZnttNJ8JJJlZfsDX+XYSx5iyMcWpVq/o/MQAB7+yRim3T6SXY+Zxf5n\nPbXWsscZmk8lrofwo+S+HzAWeJpC43RnYAqwV0+LrEc+b8GycuDZU2ld3pvZT20KwNZj5pVdf80v\n9uLy/mc9BcC020cy7faRAA4DS6XTLqOIOCAiDgDeAnaNiLERsRswBnijWgXWKnchWaVtss0iPvpw\n1W+02U9tyuynhq7zdqRVoVDkMLA00gwqj4yIZ4sLETED2CG7kuqHxxaskt59bRB9+7eu9lzf/q38\n/S/DmTB4fOqB4ohCt9Gay6VjCmYdSRMIz0j6uaT9k9vPgGeyLqyeOBSsEh68fDda1mujb/8VQNC3\n/wpa1mvjwct3A+Drm4xf7Uv94Z+M4ZFrV7/QYPH54pjBnl+YwYabLWHqbSN5+CdjaG/v+H1mkC4Q\nvgQ8B4xPbs8nz1kJtxasp154cAS3nn8Ai+atz/qDl7No3vrcev4BvPDgiJXrfH2TwqGqxS/95Uv6\nrPaLX4L1Bq5g12Nm8emvPMVHS/rw/tsD2XCzJfRdfwV/vqrj95lBimsq15JaOMooDQ84WzV1NE9S\n8QikYoth6m0jV44heIC5+VTsmsqSxkm6X9LfJb1cvFWmzMbk1oJVU0fnMRS/7IsDzKVf/g4D60ya\n2U5/AXydwlnKbV2sayU8g6pVS2czq645wAyFZYeCdSTNGML7EXFPRMyLiHeLt8wraxBuLVi1TRg8\nnvb2tQeYz/nTTex6zCym3T5ytaOOzIrStBAeknQJcDuwvPhkHmcq1zO3FqyarvmXozjw7Kns9cUZ\njD3+eQYOWb6y+2jw1u/zwYIBbiHYWtIEwh7JfemARAAHVr6cxuaznK0ailNgtC7vzaK5/Rm89RLa\nVoiWvoEEOx/xssPAOpTmEpoHVKOQZuLWgmWpOAXGRx/2Yeh2S2hvFwqxbHEv+m2wwmFgnUp7Cc3D\ngB1Z/RKaF2VVVDNwa8Gyssk2i/hgwXoA9OnbRltbYRyrpW9bMk9SsP7g5WyyTbpJ9Kx5pDnsdBJw\nPPA1CpPbHQdsk3FdTcMDzlZppVNgrPioN+oFvRS0flS4xFrf/q28+9qgil7W0xpDmqOM9o6ILwDv\nRcT3KMxy+slsy2ouPhLJKql0CozFc/vTq1eg3u0smtd/rekwKnlZT6t/qS6hmdwvlbQFsALYPLuS\nmpdDwSqhdAqM3n3g7VkbM3fWJvRuocPpMGDdL9JjjSnNGMJdySU0L6Fw+cwAfp5pVU3MYwtWCS88\nOGKtL/2udHZymzWPNC2Ei5NLaN5GYexge+A/si3LzPLkFkNzStNCeBTYFSAilgPLJU0rPmfZ8KGp\nlje3GJpPpy0ESZtJ2g3oL2mMpF2T2/7AgKpV2MQ82Gy1YtKxR/HOK4PyLsMyVq6F8DngVGA4hesr\nF7+ZFgPfyrYsK+XWguWtOFB94NlT+eR+r+ddjmWky+shSPp8Mn6Qu3q5HkKWHAxWS9yVVB8qdj0E\nYLikQSr4uaRpkg6qQI3WDe5CslriwefGkiYQTouIRcBBwCbAKcB/VWLnkg6WNEvSS5IuqMQ2m4HH\nFqyWFE9uczjUvzSBUPzmORS4LiKeK3mu2yT1Bn4CHAKMAk6QNKqn220mDgWrNQ6G+pYmEKZKuo9C\nINwraQOgvQL73h14KSJejoiPgJuAoyqw3abi1oLVIodCfUpzHsLpwGjg5YhYKmkT4EsV2PeWwJyS\n5ddZde0FW0c+Eslqjc9jqD9lz0MAiIj2iJgWEQuT5Xcj4pnSdbIk6QxJUyRNmZ/1zuqcWwtWq9xi\nqA/luozuTvH+NOt05g1gq5Ll4clzq4mIayJibESMHdqDnTUTh4LVIo8v1L5yXUa7SCp3BQ0BPbnC\nxpPAJyR9jEIQ/CtwYg+2ZyXchWS1yl1JtavTFkJE9I6IQWVuG0TElt3dcUS0Al8F7gVmAjcnRzBZ\nhbgLyWqdWwy1Jc1RRpmJiLsj4pMRsW1E/CDPWhqZQ8FqmUOhdqS6prLVP19nwWqZu5FqQ64tBKs+\ntxas1nnwOT+pWgjJWcXDStePiNlZFWXZcmvB6kExFNxiqJ4uA0HS14DvAnNZdYZyADtnWJdVgY9E\nsnrgYKieNF1G44GREbFjRPxTcnMYNAgfiWT1wt1I2UvTZTQHeD/rQixfbi1YPfDgc7Y6DQRJ5yQP\nXwYelvRHYHnx9Yj4cca1WZV5bMHqibuSKq9cl9EGyW02cD/Qt+S5gdmXZnlxF5LVE3clVU6nLYSI\n+B6ApOMi4pbS1yQdl3Vhli+3FqyeuCupMtIMKl+Y8jlrQG4tWL1xi6H7yo0hHELhojhbSrq85KVB\nQGvWhVntcGvB6o3HF7qn3FFGbwJTgSOT+6LFwNezLMpqk49EsnrjrqR1U24M4WngaUk3RMSKKtZk\nNcytBatXEwaPdyh0oVyX0bMUzkhGWrsf2SenNTe3FqweuSupvHJdRocn92cl99cn9yeDvwnMrQWr\nXw6Gjimi/P/Mkp6KiDFrPDctInbNtLIOjJViSrV3aqk4FKyeNXowTBg8YWpEjO1qvTRTV0jSuIiY\nnCzsjafNtjW4tWD1zIPPBWm+2E8HrpL0qqTXgKuA07Ity+qVz1uwetfM5zF02UKIiKnALpI2TJY9\n0Z2V5daC1btmHWMod5TRyRHxm5JJ7orPA57czrrmI5Gs3jXboarlWgjrJ/cbVKMQa0xuLVi9a6bx\nhXInpv00efjDiFhWpXqsQbm1YI2g0VsMaY4ymiFpLvDX5PY3jyNYd7i1YI2gkVsMXR5lFBHbAScA\nzwKHUZjOYnrWhVnj8pFI1iga7YikLlsIkoYD44B9gV2A54C/ZVyXNTh3IVmjaKQjktJ0Gc0GngT+\nT0ScmXE91kTchWSNpBG6ktKcmDYGuA44UdKjkq6TdHrGdVkTcReSNZp67UpKc2La05L+AfyDQrfR\nycCngV9kXJs1EbcWrNHUY4uhyxaCpCnAo8DRwExgv4jYJuvCrDm5tWCNqF5aDGnGEA6JiPmZV2KW\ncGvBGlE9DD6n6TJyGFgufCSSNaJa7kryNNZW0wK5G8kaVq11JTkQrC44FKxRTRg8vmaCodxsp8eU\ne2NE3F75csw657EFa2S1MMZQbgzhiDKvBdDtQJB0HDAR2AHYPcJXxrT0PLZgjSzPYCg32+mXMtzv\nDOAY4KddrWjWEbcWrNHlMbNqmsNOkXQYsCPQr/hcRFzU3Z1GxMxku93dhBng1oI1tmofkZTmxLRJ\nwPHA1wABxwE+Mc1qho9EsmZQjcHnNC2EvSNiZ0nPRMT3JP0IuKerN0l6ANisg5e+HRG/T1ugpDOA\nMwC2Tvsma0puLVgzyHKMIU0gfJjcL5W0BfAusHlXb4qIf+5JYSXbuQa4BmCs5P/brSyPLVizyCIY\n0pyHcJekjYBLgGnAq8BvK1aBWQbchWTNopLdSIoo/0tK0noRsbz4mMLA8rLic93aqXQ0cAUwFFgI\nTI+Iz3X1vrGSj0+1debWgjWTjloMEwZPmBoRY7t6b5oWwqPFBxGxPLme8qNl1u9SRNwREcMjYr2I\nGJYmDMy6y60FayY9aTGUO1N5M2BLoL+kMbDy/6pBwIBu79EsBx5bsGbS3fGFTruMJH0ROBUYC5T2\n1CwCfp3H1BXuMrKeciBYc1KqLqNyZyr/Gvi1pM9HxG0Vrc0sJz401axzacYQJkv6haR7ACSN8jWV\nrZ75RDazjqUJhF8B9wJbJMt/ByZkVpFZlTgUzFaXJhCGRMTNQDtARLQCbZlWZVYlbi2YrZImED6Q\ntAmFKa+RtCfwfqZVmVWZQ8Es3dQV5wB3AttKmkzhZLJjM63KLAc+NNWaXZeBEBHTJH0aGEnhXIRZ\nEbEi88rMcuIjkaxZdRkIkvoBXwH2odBt9FdJkyJiWdbFmeXFrQVrRmm6jK4DFlOYewjgROB6CtdF\nMGtobi1YM0kTCDtFxKiS5YckPZ9VQWa1xq0FaxZpjjKalhxZBICkPVh9KguzpuAjkazRpWkh7AY8\nIml2srw1MEvSs0BExM6ZVWdWY9xasEaWJhAOzrwKszrjsQVrRGkOO32tGoWY1Ru3FqzRpBlDMLMy\nPLZgjcKBYFYBnhPJGoEDwayCHApWzxwIZhXm1oLVKweCWUYcClZvHAhmGXIoWD1xIJhlzF1IVi8c\nCGZV4lCwWudAMKsitxasljkQzHLgULBa5EAwy4lbC1ZrHAhmOXMoWK1wIJjVALcWrBY4EMxqiEPB\n8uRAMKsxbi1YXhwIZjXKoWDV5kAwq2FuLVg1ORDM6oBDwaohl0CQdImkFyQ9I+kOSRvlUYdZPXFr\nwbKWVwvhfmCniNgZ+DtwYU51mNUdh4JlJZdAiIj7IqI1WXwMGJ5HHWb1yq0Fy0ItjCGcBtyTdxFm\n9cihYJXUktWGJT0AbNbBS9+OiN8n63wbaAVuKLOdM4AzALbOoE6zelcMBRE5V2L1ThH5/COSdCrw\nZeAzEbE0zXvGSjEl06rM6ptDwTqmqRExtqu1MmshlCPpYOAbwKfThoGZdc2tBeuJvMYQrgQ2AO6X\nNF3SpJzqMGtIHluw7silhRAR2+WxX7Nm4taCrataOMrIzDLk1oKl5UAwawI+b8HScCCYNRGHgpXj\nQDBrMm4tWGccCGZNyqFga3IgmDUxh4KVciCYNTl3IVmRA8HMALcWzIFgZiXcWmhuDgQzW4tDoTk5\nEMysQ24tNB8HgpmV5VBoHg4EM+uSWwvNwYFgZqk5FBpbbldM6w5J84HX8q6jxBDgnbyLyFEzf/5m\n/uzQ3J+/Hj/7NhExtKuV6ioQao2kKWkuS9eomvnzN/Nnh+b+/I382d1lZGZmgAPBzMwSDoSeuSbv\nAnLWzJ+/mT87NPfnb9jP7jEEMzMD3EIwM7OEA6GHJF0i6QVJz0i6Q9JGeddUTZKOk/ScpHZJDXnk\nxZokHSxplqSXJF2Qdz3VJOmXkuZJmpF3LdUmaStJD0l6Pvk3Pz7vmirNgdBz9wM7RcTOwN+BC3Ou\np9pmAMcAf8m7kGqQ1Bv4CXAIMAo4QdKofKuqqmuBg/MuIietwLkRMQrYEzir0f7bOxB6KCLui4jW\nZPExYHie9VRbRMyMiFl511FFuwMvRcTLEfERcBNwVM41VU1E/AVYkHcdeYiItyJiWvJ4MTAT2DLf\nqirLgVBZpwH35F2EZWpLYE7J8us02JeCdU3SCGAM8Hi+lVRWS94F1ANJDwCbdfDStyPi98k636bQ\npLyhmrVVQ5rPb9YsJA0EbgMmRMSivOupJAdCChHxz+Vel3QqcDjwmWjA43i7+vxN5g1gq5Ll4clz\n1gQk9aEQBjdExO1511Np7jLqIUkHA98AjoyIpXnXY5l7EviEpI9J6gv8K3BnzjVZFUgS8AtgZkT8\nOO96suBA6LkrgQ2A+yVNlzQp74KqSdLRkl4H9gL+KOnevGvKUnIAwVeBeykMKt4cEc/lW1X1SPot\n8CgwUtLrkk7Pu6YqGgecAhyY/L8+XdKheRdVST5T2czMALcQzMws4UAwMzPAgWBmZgkHgpmZAQ4E\nMzNLOBCsaiSdKmmLFOtdK+nYtM9XoK5vlTwekWYmz6SWVySdWWad0ZU8LDH5+13Zw208XJyVVtLd\nPZ2dV9L+ku5KHh+fzAB7V0+2aflxIFg1nQp0GQg5+FbXq3To/Igod97JaCC349QllZ2JICIOjYiF\nldpfRPwO+LdKbc+qz4Fg3ZL8kn5B0g2SZkq6VdKA5LXdJP1Z0lRJ90raPPllPxa4ITmhp7+k70h6\nUtIMSdckZ4Km3f9a+0ief1jSDyU9IenvkvZNnh8g6eZkLvs7JD0uaayk/wL6JzUV56HqLelnyZz3\n90nqn6Ke45LP8bSkvyRnMV8EHJ9s+3hJu0t6VNJTkh6RNDJ576mSbpf0P5JelHRxyXa/lHyOJyic\nGFV8/ojkMzwl6QFJw5LnJ0q6XtJk4Prk73xT8t/oDqB/yTZelTRE0pklJ1q9Iumh5PWDknqnSbpF\nhTl8itf1ngvnAAAD10lEQVSDeEHSNApTn1ujiAjffFvnGzACCGBcsvxL4DygD/AIMDR5/njgl8nj\nh4GxJdsYXPL4euCI5PG1wLEd7PNa4NgU+/hR8vhQ4IHk8XnAT5PHO1GYiHBssrxkjc/VCoxOlm8G\nTu6slpLlZ4Etk8cbJfenAleWrDMIaEke/zNwW8l6LwMbAv2A1yjMl7Q5MBsYCvQFJhe3B2zMqhNL\n/63kM08EpgL9k+VzSv42O6/xuV8FhpTU1wf4K3AEMITCNS7WT177JvCdpL45wCcAJX+fu0q2sX/p\nsm/1dfPkdtYTcyJicvL4N8DZwP9Q+MK9P/nB3xt4q5P3HyDpG8AAYDDwHPCHFPsd2cU+ipOOTaXw\nBQ+wD3AZQETMkPRMme2/EhHTO9hGOZOBayXdXLL/NW0I/FrSJyiEaZ+S1/4UEe8DSHoe2IbCl/LD\nETE/ef53wCeT9YcDv0taRn2BV0q2dWdEfJg83g+4HCAinunic18GPBgRf5B0OIULAE1O/sZ9KUxZ\nsT2Fv8+LSU2/Ac4os02rIw4E64k15z0JCr8an4uIvcq9UVI/4CoKv1bnSJpI4ddnGl3tY3ly30b3\n/o0vL3ncRkk3S2ci4kxJewCHAVMl7dbBat8HHoqIo1WYT//hMvvsqu4rgB9HxJ2S9qfQMij6oKt6\n16TCjL3bUJinCQp/4/sj4oQ11hu9rtu2+uExBOuJrSUVv5RPBP4GzAKGFp+X1EfSjsk6iylMBAir\nvvzfSfqm1+XooXL76Mxk4F+S9UcB/1Ty2goVpjXuNknbRsTjEfEdYD6FLp/SzwuFFkJxquxTU2z2\nceDTkjZJ6juuk219scw2/kLhvw2SdqLQbbRm7btR6FI7OSLak6cfA8ZJ2i5ZZ31JnwReAEZI2jZZ\n74Q1t2f1y4FgPTGLwnVlZ1Lo0746CpeVPBb4oaSngenA3sn61wKTJE2n8Iv4ZxSuyXwvhWmlU+li\nH525ikKIPA/8B4XuqfeT164BnikZVO6OSyQ9q8Ihq48ATwMPAaOKg8rAxcB/SnqKFC2XiHiLwi//\nRykE2sySlycCt0iaCrxTZjNXAwOT/0YXUegCW9NXKXTZPZTU+vOkm+pU4LdJN9OjwPYRsYxCF9Ef\nk0HleV19Dqsfnu3UuiXp8rgrInbKuZRUJPUG+kTEsuTX7QPAyCRcurO9ayl8/lsrWGbdS7qvzouI\nw/OuxdadxxCsWQyg8Au4D4X+8a90NwwS7wPflzQkyp+L0DSSVtB36bgVYnXALQQzMwM8hmBmZgkH\ngpmZAQ4EMzNLOBDMzAxwIJiZWcKBYGZmAPx/WPXT3D6XX9IAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x111428690>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_decision_regions(X_combined_std, y_combined, classifier=svm,\n",
    "                     test_idx=range(105, 150))\n",
    "plt.xlabel('petal length [standardized]')\n",
    "plt.ylabel('petal width [standardized]')\n",
    "plt.legend(loc='upper left')\n",
    "plt.show()\n",
    "           "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note** 逻辑斯蒂回归 VS SVM\n",
    "\n",
    "在解决现实的分类问题时，线性逻辑斯蒂回归和线性SVM通常效果近似。逻辑回归目标是最大化训练集的条件似然,使得她更易受奇异值影响。SVM只关心那些离决策界最近的点(即，支持向量)。另一方面，逻辑斯蒂回归的优点是易于实现，特别是并行化实现。此外，面对流式数据，逻辑斯蒂回归的易于更新的特点也很明显。\n",
    "\n",
    "\n",
    "### scikit-learn中不同的实现方式\n",
    "\n",
    "前面我们用到的sklearn中的Perceptron和LogisticRegression类的实现都使用了LIBLINEAR库，LIBLINEAR是由国立台湾大学开发的一个高度优化过的C/C++库。同样，sklearn中的SVC类利用了国立台湾大学开发的LIBSVM库。\n",
    "\n",
    "\n",
    "调用LIBLINEAR和LIBSVM而不是用Python自己实现的优点是训练模型速度很快，毕竟是优化过的代码。可是，有时候数据集很大，不能一次读入内存，针对这个问题，sklearn也实现了SGDClassifier类，使用提供的partial_fit方法能够支持在线学习。SGDClassifier利用随机梯度下降算法学习参数。我们可以调用这一个类初始化随机梯度下降版本的感知机、逻辑斯蒂回归和SVM。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.linear_model import SGDClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "ppn = SGDClassifier(loss='perceptron')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "lr = SGDClassifier(loss='log')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "svm = SGDClassifier(loss='hinge')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 使用核SVM解决非线性\n",
    "\n",
    "\n",
    "SVM之所以受欢迎度这么高，另一个重要的原因是它很容易核化(kernelized)，能够解决非线性分类问题。在讨论核SVM细节之前，我们先自己创造一个非线性数据集，看看他长什么样子。\n",
    "\n",
    "使用下面的代码，我们将创造一个简单的数据集，其中100个样本是正类，100个样本是负类。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "np.random.seed(0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X_xor = np.random.randn(200, 2)\n",
    "y_xor = np.logical_xor(X_xor[:, 0]>0, X_xor[:, 1]>0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "y_xor = np.where(y_xor, 1, -1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXYAAAD8CAYAAABjAo9vAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJztnX2IHdd5xp93HcWCZlfyZ91ko8qlIdQ1YYuUKiIQt0lo\n3ZAmpB9Q/5HiuiAKLrqWUtIGg7WmFFISJK2bQAh1yD/BbSFfJm3zRZOWQmq6Ktvi+CMkhdJtEuqo\n1kqQurG1b/8Yje/c2fk4M+ecmXPOPD8YpHv3zsyZ2b3PvOc573mPqCoIIYSkw9LYDSCEEOIWCjsh\nhCQGhZ0QQhKDwk4IIYlBYSeEkMSgsBNCSGJQ2AkhJDEo7IQQkhgUdkIISYxXjHHSm2++WQ8fPjzG\nqQkhJFouXLjwA1W9pe1zowj74cOHsbm5OcapCSEkWkTkP0w+RyuGEEISg8JOCCGJQWEnhJDEGMVj\nJ4SQoXjxxRexvb2NF154YeymGLN//36srq5i3759vfansBNCkmZ7exvLy8s4fPgwRGTs5rSiqrh4\n8SK2t7dx++239zoGrRgyCOX1XLi+CxmKF154ATfddFMUog4AIoKbbrrJqodBYSfeWV8HTp2ai7lq\n9np9fcxWkSkRi6jn2LaXwk68ogpcugRsbMzF/dSp7PWlS/4id/YQyJShsBOviADnzgGzWSbmS0vZ\nv7NZ9r6PQIo9BBIa9913H2699Vbceeedg5yPwk68k4t7EV+iPlYPgaSDj97evffeiy9+8Yv2BzKE\nwk68k4trkWJE7ZIxeggkHXz19t7ylrfgxhtvtG2eMRR24pVixDybAbu7c9H1Le5FKOqkjZR6e8xj\nJ14RAQ4eXIyYc9E9eNCfHVPVQ6C4kyaKf5sbG9kGxNnbo7AT76yvZ2KbfzHyL5BPUS/aL/lrIL4v\nKBmW/G8z/3sB4vyboRVDBqH8xfD1RanrIcxm/noIJB2GHA/yCYWdJMf6+mKUlYs70x1JEz7Hg+65\n5x4cP34czz77LFZXV/Hoo4+6a3gFtGJIkgzVQyDp4HM86LHHHnPTSEMo7IRUsbICXLmy9/3lZeDy\n5eHbQwZhyPEgn9CKIaSKKlFvep8kQwq9PQo7IYQkBoWdEEISg8JOSFdE9m4rK2O3ipCXsRZ2EXmt\niHxNRJ4SkW+KyMxFwwiJCnrvJCBcROwvAXifqt4B4E0A7heROxwcl5DxWF4euwUkQZ555hkcP34c\n119/PT784Q97O491uqOqfg/A9679/4qIPA3gNQCesj02IaNRl9IYY4oECYYbb7wRjzzyCD73uc95\nPY9Tj11EDgP4OQBPVPzshIhsisjmc8895/K0hBDihpUVr2Mot956K974xjdi3759To5XhzNhF5FX\nAfg0gAdUdU+4o6ofV9Wjqnr0lltucXVaQghxRyLzF5wIu4jsQybqn1LVz7g4JiFBUue905MnAeEi\nK0YAPArgaVU9a98kQgLm8uVsznl5Y5kBUsNHP/pRrK2tYW1tDd/97ncHOaeLiP3NAN4L4K0isnVt\ne4eD4xJCSPTcf//92NrawtbWFl796lcPck4XWTH/CICpAoQQ0sL3v/99HD16FJcvX8bS0hLOnz+P\np556CiuOJ7ixuiMhhOQsL9dX9XTAbbfdhu3tbSfHaoIlBQhpw3MKHAmIRMZQKOyEtJFICpx3+AAM\nBgo7IS6gqAX9ANTIFi21bS+FnZCcOnE2IWBRmzr79+/HxYsXoxF3VcXFixexf//+3sfg4CkhORTh\nJFldXcX29jZiKmWyf/9+rK6u9t6fwk4ISZp9+/bh9ttvH7sZg0Irhjin3OONpAfcHZYRIIFCYSdO\nWV8HTp2ai7lq9np9fcxWOSCBFDjvsI5OMFDYiTNUgUuXgI2NubifOpW9vnTJLnIPvhdAUUsmBzwF\n6LETZ4gA585l/9/YyDYAmM2y9/uuUbG+nj0Y8mPkD4yDBx33BGxmHVK8SEAwYidOKYp7jo2o++wF\n7IERJ0kECjtxSi68RYqee1fyB8Vslon50lL2r20vgJCUobATZxSj6dkM2N2dC7ILcS9CUSekHgo7\ncYZI5nsXo+k82j540M6OMekFBD/A6gOWMiAVcPCUOGV9PRPUXMRzcbcV9aL9kr8G5scebIA1BFZW\n2mfJchbtpKGwk5cpCnLVa1PK+9hYJnW9AGDeCygOsAKL4j+b9b+OYDARckIKyBiFcY4ePaqbm5uD\nnzdlbEW5b8Tr6mHQRtt5ipF9TjIDrDbdHZIUInJBVY+2fY4eewLYzvbsm1I45CzTtl4AB1gboA8/\nOSjskeMqz7trSuGg+eUGuE6zTAqWFJ4eqjr4duTIESXu2N1Vnc0WZ9XMZtn7bZw5M//s7u7iMdr2\ntzmvS4rtyM9ffh011dOmmrflZbP9SVQA2FQDjeXgaQLkNkTRXzaxIYpRd1Vk+8ADwPnz9cfpe17X\nmAywToLlZc6SJQCYFZMEdTZEm8jmAqgKPPLI/P2TJ7N/H3mkOV2x73l94DrNMiiaathQyEkF9Ngj\nx3a2p0gWlRc5fz7bmiYW+Zpl2of8XHk7y6+jhzVsSEcYsUeOrQ3RFHU3RbxN5z1wYJgUSGBiE5P6\nYlO1kkQJhT0B+toQJrM6889VCXXVeQ8cAHZ25u+rZl79DTfMhdaV0LdNTNrdzTJ8yu2eHIzsJwet\nmEToM9vTpLZLW656WfB3dhbtmOPHM6/++efnDoKrXPemyo8HDgCnT3fMsWe+N0kFk9QZ1xvTHcOi\nnA6Yv+6TRliVAgmonjzpLw2xnKZ59WrP9EdXaYHLy+0piIT0AIbpjhT2iVMn6sXXXXPVy0J78mS3\n/bu2v6p9RXE3Pq8rYXdxHD4cSAWmwu7EihGRT4jIf4vIky6OR4bBpCRA16n6+TGacJWGmJ+rKjPn\n9Gng7Fk/5x0EzhYlFrjy2D8J4G5HxyIDoGpWEqBKqOvSGauE9uTJxRz5pv270jRGkHvsPs5LHMJx\nDT+YhPUmG4DDAJ40+SytmDBos1n6eOzlEgW5DXPsmF+PvYg3j93UHnFhxbiyhUJnKtfpCIRWUkBE\nTgA4AQCHDh0a6rSkgbaSAH1y5MspkDfckEXteWkCH1P9y8dZWuqZ29+W7017hESCs3rsInIYwBdU\n9c62z/atx14UjKrXpBtF6ySnqqKj7X0f6/fm/LxNOxe/R3ULYzQ9IMrlAUzPFTtTuU5HJFePfcja\n31OgaeCx7EXbrojkckWlGM7bWALAJOpv8pc5W5QYEMXM0+JAH5Dg0meeqYtcp1ARMcpeXpO1w1mk\nxAAnwi4ijwH4BQA3i8g2gDOq+qiLY2fHn4vOxsZc4JNZ+gz+BKitlkrxPElVRATryEQB69h4wYkV\no6r3qOpPqOo+VV11Keo5KS995tJmKlooqtlU/qaUxtHsCs+YpnN2ok5sKEL9YeVKL0TjsedfzCIp\n5CW7FKDyAyLn2DHzJe9SoVxH5srSCs5vCBSC8xsCWeqRL00RIrFgkhPpeuuax5760mculphrukfl\nKf2x368u5OUNgsmXNsmFD6WtJDgwZEkB35hUIYwZFzZTXaXDfDWkIin0dExQBf7v+hUoAvoDMYn6\nafkQS6IQdiCzGcqTZ86dS2MQzJXNVPWAALIp/a5XOSrvG9qDIr+n+1+McPLQVCwflhPwRjTCDqQ5\n0Ff01G3Ft+oB8cQTWdTusqcTw5yCvJdHAoYzeb0RlbCniCubqe4B8cQTe89n09Pxkm3iiZAeNIQM\nSRQTlFLHRT55lwlHNj2d5OYU9PGtm0oGpGaXkChxViumC31rxZB2fE10Kh9LdXE90d3dQEXdRy2S\nIeubuKg7EyqsE9OZ5GrFEDN8jUMUfXXVbIHqIsFm2sSeYdLkQ4fkUXMgNChoxZBWir56Lt754hl5\nOmWxjk+QkTvxS5+HDMsJeIPCTlop++o5eZ31/DNBzilwGdXW2SJTp29UHoNdFCn02IkxTb66Sy+/\nrQ2dxhBc+rgmF+j6+9T3pla1w9egb1sbg/To4oQeO3FK2ySqIUR9sPz5Or+4jdAthJA8eeKVZIQ9\n9JmQObG0s4jLSVT58Zpe1+1Tlz///PN7q1pa0UfoqmaG1j0gugwqNg3+xj4wTLyRhMduW3fbZ4qg\ny3aOhctFOfreg7r8+WPHFj8X1D1tekCYPjxC9KG7jDXwITMOJpXCXG9dqzs2YVv58cyZxc/l+585\nY3buptcu2xkCXa63bn/be5BXa8y3vHJl7fFMqimWaaqu2LSVj9n2+aFx1ZYu94Q4BYbVHaMXdtX+\nZW9thKbPA8FFeV5X2Iq0zXn73oOqfU+e3FuW2Pqe9hX2spD1Fb0+DyPb63J1HJftJXuYlLCr7o3k\nTL/YXYSmLOJ9Hgh92+kSm16KC/rcg0HrzbcJoG9h9xUBu3pgMEIfjUkJu20kbCI0ZTG8elV1ba3b\nOUOI2IeyhOp6BHX34OrV9mNWPZBOnlQ9dszxPU1V2F0RevsSZjLCfvXqojCVX3exY+qEoUkMTSPF\nkDz2Oktjd3dRNPtS1yN46KH5edfWFn9Xa2vZz03aXvx/q8feBxthN936evy+bJouUNhHw1TYo86K\nyTMsDhzIMjbOngVOn56/bsvYUF1M4zt3bv4amGdu1GVkrK0BW1vz4506VT+l3mVmiS35uYuzSIHs\nfuT3b2enX2aJ6jwtEVi8p7NZllCR37fTp7Pf2d//ffb6rruy/ZvuRblK5Q03VN/TAweGyXQyRtXN\ncULIRWcpgPAxUX/Xm4uIvRydVUXuJnTxm8uWTZ9IcaxBy/I5y750HjUX/20ba2h63dQLKv6uXNgn\n5V5G3jPI/wY6jyG0RaR1UbOrSLZvL8CWEHoDpBFMwYpx5VmbiG3VuXI7ofjzoQYg+1IedKwS+CZr\npM7nLl53bukUj1e+p1evLv7c9EFcR96uKnunsz3TVzxdiW6TwPoUdloswTMJYVcdJsvE1B+PIRdd\ndVGcywLb9JCsuu584LLo0bcNaD700N6BZ1OPvYqq3lvXge0F+kauQwgjhX3STELYu0bsNjbI2CmC\nrslFuGoQuNwbKe9XtU8u7MUeQP5eUXRfemnR8imKcN05Ta+n7lq8PPC72jGu7AwK+6RJXti7Zpm4\nEOYQ/HFXFO9XObpte0iWe0llO+fYsbmoF8+V32sfEXtVu3pH7CZ0EXWX4ujTB6ewB4+psEebFdMl\ny0S1OVND1SxjwtfqRGNQvH95Papihs/a2t7MEiC7V+Uqj2W+8Y35OfJ/i9lCDz8MnDkDXHfdfJ8L\nFxZLApuSZ0blGVFF1tay454+3bIQSMhrmIbcNhIuJurvenPtsTe9Lr4/9uSgEOkyD6CqV1Q1+DrU\nRK2qXkf+7803772m2t5Zn0h1qIh9iCi6zVZiVkwwIEUrxtYKGWKgNUa62FTFzxaF/dixdjus7OtX\nefA24l58SLz44uI1NB43FGHv6tu7ghZMNJgKuxMrRkTuBrAB4DoAf66qH3Rx3CIuSvNWLRTBNTqz\n+6dab500ffaGG+ZL5DXZYcXf38GD87VSH37YbqKWSGYZ5exgBSsbV4AN4Hz+5sOAxGBddJ1ktLIS\n/jWRcTBR/6YNmZh/B8BPAXglgH8FcEfTPl0jdtvp+CFN50+RLhOWyhZOMfrvQzm10Xn0XXE9quon\nK6ZnL8B6UD/yiD2lpIY2MJQVA+A4gC8VXn8AwAea9uljxdj6stGlK4Y2C9CyPT7GOKo8dtfCXvd3\nUvX3ZC2QPYS9qS5P+V71Om/gxPK9dvXwGVLYfwOZ/ZK/fi+Aj1R87gSATQCbhw4d6nVRth75oE92\nW2H28WWzaZOD9vgY4yjOOO0t7DX3ZXd5uXYiWlUPcAcef+c1W11PtNOs6JYHW5GQouFYeuIuHz7B\nCXtxGyNiHxyf0dsYbbJsj8/fXzGTZ6h2ermeHsJelZlUrvXTKnY1D7YXXrkcfDQcui64fvgkZcXE\n8mReICVh7+InV0Snzn5/Ne34333LLx/LxX0z7Vk474H0KC5WJewuiqwN/Z2z6RmEnu3m8uEzpLC/\nAsC/A7i9MHj6s0379InYY/HSXiYlYe8oNlU4+f01nPPlYw80FjBopNhgFdXNIyjXAOrTrqGu0Xbd\n4ZAj9hxXD5/BhD07F94B4FvXsmMebPv8WHnsgxKbsDcJoQNhV+35+zOMYl38bZhGqWP2IKuEDNg7\nJ8CqCFrpfC4EyeR6ut7LWHryUUbsfTYfa54Gh60w+8iK6SvQjoTda5sdYRo9jtmDLIttsS5PMf2z\n7eFkcp4houG685QLwlWdN/SefLQee59tEsIeWrpiU5uGEnbTe+J7IYsWTKP/MXqQVSJYFHbVeQ36\n/L286FqeAmkifmN47MVrKl9DU5tD78lHmRXTZ5uEsMfCkMJuuk/X87nuJQRKF6vIhW00VDRc9bDq\nnNkTOK4ePhR2YoaJWFpmxXQ6V9vnup4zMcpiaxKN21gqvqPhpgePq3GClDAV9h6FUkkwrKzMV9su\nbnkdXldcvrxXTkNZuDiGGjAOWV9frJe0s5OVW97ZyV7n5agvXcpeA4s1fHJMayT5LlVdV357NgPe\n9a7Fz7KuUwdM1N/1xojdES6siSGXgfMVsbu4noFwHQGbROMxpASW2+J6wfNUACN2skBddA9Uy2Ws\nUXBdhcSK9/OItvi66r0uNO2/vp5F1Pl7eYR95kz/c7ZF48UofjYDdnezfzc2FtsyNuWFcfLFUUJu\nc8hQ2KdCB8HzRp19U35/AJunSmSPH8+2svCalIWuO2a+v+p8Fa/8M7ngPv54Jl59zpl/vkixDU1W\nR58yyUMQY5uDwySsd73RinFEFzvEhW3j83iuzmfwuaoBu7ZFuNssAJPsE9fZH10yXkJPCawixjb7\nBsyKmQCuhL2PL+1S2E3O71DYVevzwctT9Lv4uqZ+d/Hntl5y6BN0iFtMhV2yzw7L0aNHdXNzc/Dz\nJkeXhY5d9l9zq6Tp3K7alv99mh7P5FiFl8UFtHM7pPxel1tXdcwqvztnNssW4i4u7N3nnGWPmnZF\nmojIBVU92vY5euwxU5WGqOp/4PPKlfZzu/b0Ta/V0MfPRbbIAw9kW5Eug3VVx8z3L4p6eUDwyJH+\n5wT8pySSCDEJ611vtGJGoMnuaLIv2mwN2+n/bft7YCyPvWybmNZ16XuNTa9JnGDIxayjpYtdEDtN\n12MT4tlm1QyZlXONqqyL8+eBJ57Ift62MLfpMcv7lxcCX1rKJuHcdVf9Pn2wXfidxM+0PfYOfmzS\n9FGQ/P7Y7Guyv8ffQ5U3XW5SV7+6j9/t0iMvWz7nzu19TasmXkw99mlH7KHR1oPw1cNYXh4lcjZC\nxFsPysSb7iqCffxulx55Merf2JgP1FLUpwUj9jrGiNjb2jN0e00eJF2VoksWS5Ep9aAcoA3ZOcXP\nMJsmLpgVQ+yxzbqp27dY3oA4pyk7J6dpliyJHwp7HT6rJaaEaZmAIqHaPiNR7ozYdE6a0iqLqZd1\n5Q2KVSFJvEzbYzf1lilE9QyVPVSM7hPKWnKdwWKSnQPQh0+daXvsZcb23EPz2H3hQjliut4afGaw\nmPjnJj48CQt67DHSZF80feNCWfSCdKJYtXBjIxNZV2mJbZk2Jj48iRcKe0iUByub6DOYGQOm158I\nNqsb9cXEhydxQ2GfKkMtq1dFW8/EJGNmjHZ7YIzImfXO02fag6dl6gZTU7Q6hlx4oy4f3iURDnA3\neeyA38i9XN4gF3eKehpQ2IukZGmERISiC/ifwGOaweILkxmvnMQUJ8yKcUmXyNQkZc9nFsyQGTYu\ns2AGaveQhbRCFU8WEwsPZsX0wdZ37hKZmny2z+QfF4zlvTeR/24GYOgJPCHWU+ckprhhxF7ENhrs\n+o0c89thW5XR97n64PB+FoUsxyQNMdTouw997wHxh2nETmEvMiVh7zOgOZawm8wQ9jAbtesEntCs\nCxcPGU5iCotBrBgR+U0R+aaI7IpI68lIQNQV+AoNkzz9XPgd2kdd0xBDsy5cFPniJKaIMVlmqW4D\n8DMAXg/g6wCOmu4X7NJ41VKXbbb79z3m0Phob9sxTc7Z9d5atNlkmbu2/fLNdom7Idvv+hjEPRhi\naTxVfRoAhH2zjC4LVqSYGx8JbRZF3zTE/HNFT3oMP9rFYhtjp2ISO5x47CLydQB/oKpGxnmwHvuU\n1kCtw8c9cFHcrKldTQ/T0t93Fx+8q0ddtF9yug42uvDFi/va+uMu20Psceaxi8hXReTJiu3dHRt0\nQkQ2RWTzueee67LrcNguLJECY9yDgdI6taMP3iUNsXisvvVXXC5+ke9bpI8/HmIqJjHAxK9p25CK\nx04WWV6u9q2Xl7vvY+uBO/LYffrgZ84sHis/15kz7fu69LTpj6cLDD12WjEp4MtC6pP+6Svls61s\ncYfrd2FR1GFjXbiwcnJCS70kbhgkj11E3gPgzwDcAuASgC1V/eW2/SjsjvE1zX4IYTcVZUfX6FI8\nfeDyoUN/PD0GyWNX1c+q6qqqXq+qP24i6oQsMGCVSVMfvPyccNCp7dS+IjZ54/THpwtrxYTAmLXR\nx6RtcNTx9ZvUIXc5gNkFF4OvhOSwbG8IDFkbfUyq1KkpjCxev6Na+U11yItZM8DeNUhtUw995M4T\nUgVrxYSAzxo1bfs3DbwC3Qdluw7ktinWwH+fPjx4n7nzZFqwbO8QhGKh2OSBN/UW+uS0Rz4XoBgp\n59iIerEX4Dp3npA6KOw2hGKhxCymrksrWD5sfQxg5j7+xkaW8VJcCi+P4AlxCYWdjIvrh4/Fw9bX\nAGZVL+Ds2UVbhrnlxCUU9hAYa6WkUAjk+k2yZvpQ1Qs4ciR7cHBVIuIDDp7a4Gti0JAMfQ2+C605\nuB6XA5jlXsDZs5mob23NPxPSBCkSNhw8JWYMHS2HMi7RgMsBzHIvYGkJuHBh8TMUdeIa5rHb4Ci3\nelRiGGCNnGLuvCpw+vTiz0+dorgTtzBityHmbJRUCcSvL1McKI1hdulYZRWIGxixk7QI+KEay+xS\nVoaMHwo7mQwhzOpsKmkQAj7LKpDhoLD7gsvsVTPSuERIUWjIs0tdrJdKxoceuy8iyP4YhRHGJbpO\n6586rssqjMWUxwko7CR5TKb1kzmuyyqMwVjll0OBwk4mQYxR6BgRZ0yZO3Wwh0aPnaSAwXhGXRQa\nqriPNSYQS+ZOExwnAFpXu/axHTlyxOnK3UFS7SRnG3FLy73e3VWdzbKXs1n165AIob3lc4R2j0zY\n3V38U4jxGsoA2FQDjWXE7osUZqUmQmxRaAgRZ8iZOybE1kNzDYuAkfgxLPxlksceQq578dxLhVGw\n3d1piJIt5XGCci5+zOLOImChE8rqSxOiLQoNKZMihcyUsfBVfjkmaMWMBfPcFxl5QperGZcuIv6m\niDNv2xTEyYbQZ/j6hsJOwsDmQedgPMOFr+0qkyW2MYFQiX2cwAZ67GORwiIdLgnkfvT1tX34uiH5\n/SQM6LGT8YlsHMHG1/Yxu3XKESexg8JO/BHROIKLGZcxzm4laUJhH4tAF4SYKi4yKZjJQkKBg6dj\nYZrpMZXyvwFM6LLJpGAmCwkJK2EXkQ8B+FUAPwLwHQC/o6qXXDSMXCMiO8OKQB5SfX1tZrKQkLDK\nihGRXwLwd6r6koj8KQCo6h+27cesmA4Eki3Si5jb3hNmshCfDJIVo6pfVtWXrr38JwCrNscjiTHB\ncQRmspAQcOmx3wfgLx0ej8ROIPYKIVOjVdhF5KsAbqv40YOq+vlrn3kQwEsAPtVwnBMATgDAoUOH\nejWWEEJIO63Crqpvb/q5iNwL4J0A3qYNhr2qfhzAx4HMY+/WzAkTQLYIISQubLNi7gbwfgB3qeoP\n3TSJLEA7gxDSEdsJSh8BsAzgKyKyJSIfc9AmQgghFlhF7Kr6064aQibCVCZcETIiLClAhmUqE65I\n0pRHE0OblkFhJ4SQDoS00lYdFHZCCDGkuNJWLu55TaBLl8KJ3FkEjBADWCqAAG5W2hoCRuyEtBBD\n15sMRwx19ynsZFgiqx8TS9ebDEcMdfdpxZBhiSylMZauNxmGWOruM2InpIUYut5kGFystDUEjNgJ\naaGu601xnyY2K20NBSN2Qhpwscg1SY/Q6+4zYiekAS55R2LEamm8vnBpPBIbzGMnITDI0niETIXQ\nu95NhF7XhLiHwk5IwnBy1TShsBOSKJxcNV04eEpIonBy1XTh4CkhEdFnEFcVWCr0zXd3KeqxwsFT\nQhKjj18eQ10T4h4KOyER0Mcv5+Sq6UKPnZAIaPLLz56ttmc4uWq60GMnJCLKfvlDDwE7O3PhzqP0\ngwfnFg0nV6UDPXZCEqPKL3/88XZ7JubJVaQfFHZCIqDOL9/aAtbWsveXlhbrhFPApws9dmLHygpw\n5cre95eXo1tUI2Sa/PIDBzKBz6GoE0bsxI4qUW96n/RmfX1RtEWygdOdncXPMeOFUNgJiYjyIOjp\n00xnJHuhFUNIpDCdkdTBdEdiR5N6MGQcBKYzTgemOxIyEZjOSMpYCbuI/LGI/JuIbInIl0Xk1a4a\nRiJhebnb+4QQ79hG7B9S1Teo6hqALwB4yEGbSExcvpz1/csbUx0JGQ0rYVfV4rf3xwDQVCWEkJGx\nzooRkT8B8NsAdgD8onWLyLTgBCdCnNOaFSMiXwVwW8WPHlTVzxc+9wEA+1X1TM1xTgA4ce3l6wE8\n26vFfrkZwA/GbsSAjH69R4AjdT+7AFxwfLrRr3dApnStwHSu9ydV9Za2DzlLdxSRQwD+RlXvdHLA\nERCRTZNUolTg9abLlK4VmN71tmGbFfO6wst3A3jGrjmEEEJssfXYPygirwewC+A/APyefZMIIYTY\nYCXsqvrrrhoSCB8fuwEDw+tNlyldKzC9621klJIChBBC/MGSAoQQkhgU9hIi8iEReeZaqYTPisjB\nsdvkCxH5TRH5pojsikiyGQUicreIPCsi3xaRPxq7PT4RkU+IyH+LyJNjt2UIROS1IvI1EXnq2t/y\nbOw2hQCFfS9fAXCnqr4BwLcAfGDk9vjkSQC/BuAfxm6IL0TkOgAfBfArAO4AcI+I3DFuq7zySQB3\nj92IAXn2eO1+AAABl0lEQVQJwPtU9Q4AbwJwf+K/XyMo7CVU9cuq+tK1l/8EYHXM9vhEVZ9W1RAn\nirnk5wF8W1X/XVV/BOAvkKXmJomq/gOA/xm7HUOhqt9T1X+59v8rAJ4G8JpxWzU+FPZm7gPwt2M3\ngljxGgD/WXi9DX7xk0REDgP4OQBPjNuS8ZnkCkomZRJE5EFk3bxPDdk215iWhCAkZkTkVQA+DeCB\nUnHCSTJJYVfVtzf9XETuBfBOAG/TyPNB2651AvwXgNcWXq9ee48kgojsQybqn1LVz4zdnhCgFVNC\nRO4G8H4A71LVH47dHmLNPwN4nYjcLiKvBPBbAB4fuU3EESIiAB4F8LSqnh27PaFAYd/LRwAsA/jK\ntZWhPjZ2g3whIu8RkW0AxwH8tYh8aew2uebaQPjvA/gSsoG1v1LVb47bKn+IyGMAvgHg9SKyLSK/\nO3abPPNmAO8F8NZr39ctEXnH2I0aG848JYSQxGDETgghiUFhJ4SQxKCwE0JIYlDYCSEkMSjshBCS\nGBR2QghJDAo7IYQkBoWdEEIS4/8BSkPNHP50OKQAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1149b0c90>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.scatter(X_xor[y_xor==1, 0], X_xor[y_xor==1, 1],\n",
    "           c='b', marker='x', label='1')\n",
    "plt.scatter(X_xor[y_xor==-1, 0], X_xor[y_xor==-1, 1],\n",
    "           c='r', marker='s', label='-1')\n",
    "plt.ylim(-3.)\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "显然，如果要用线性超平面将正负类分开是不可能的，所以前面介绍的线性逻辑斯蒂回归和线性SVM都鞭长莫及。\n",
    "\n",
    "\n",
    "核方法的idea是为了解决线性不可分数据，在原来特征基础上创造出非线性的组合，然后利用映射函数$\\phi(\\cdot)$将现有特征维度映射到更高维的特征空间，并且这个高维度特征空间能够使得原来线性不可分数据变成了线性可分的。\n",
    "\n",
    "举个例子，下图中，我们将两维的数据映射到三位特征空间，数据集也有线性不可分变成了线性可分，使用的映射为：\n",
    "\n",
    "![](https://ooo.0o0.ooo/2016/06/21/5768fdd6f1aea.png)\n",
    "\n",
    "注意看右上角子图到右下角子图的转变，高维空间中的线性决策界实际上是低维空间的非线性决策界，这个非线性决策界是线性分类器找不到的，而核方法找到了：\n",
    "\n",
    "![](https://ooo.0o0.ooo/2016/06/21/5768fe8f9b373.png)\n",
    "\n",
    "\n",
    "### 使用核技巧在高维空间找到可分超平面\n",
    "\n",
    "\n",
    "使用SVM解决非线性问题，我们通过映射函数$\\phi(\\cdot)$将训练集映射到高维特征空间，然后训练一个线性SVM模型在新特征空间将数据分类。然后，我们可以使用相同的映射函数对测试集数据分类。\n",
    "\n",
    "上面的想法很不错，但是如何构建新特征是非常困难的，尤其是数据本身就是高维数据时。因此，我们就要介绍核技巧了。由于我们不会过多涉及在训练SVM时如何求解二次规划问题，你只需要知道用$\\phi(x^{(i)})^{T}\\phi(x^{(j)})$替换$x^{(i)T}x^{(j)}$就可以了。为了免去两个点的点乘计算，\n",
    "我们定义所谓的核函数(kernel function):\n",
    "![](https://ooo.0o0.ooo/2016/06/21/576935084b78b.png)\n",
    "\n",
    "\n",
    "常用的一个核函数是Radial Basis Function kernel(RBF核)，也称为高斯核:\n",
    "\n",
    "![](https://ooo.0o0.ooo/2016/06/21/5769356d7c668.png)\n",
    "\n",
    "通常简写为：\n",
    "\n",
    "\n",
    "![](https://ooo.0o0.ooo/2016/06/21/576935b1e2fc0.png)\n",
    "\n",
    "此处，$\\gamma=\\frac{1}{2\\sigma^{2}}$,是一个要优化的自由参数。\n",
    "\n",
    "\n",
    "通俗地讲，核(kernel)可以被解释为两个样本之间的相似形函数。高斯核中e的指数范围<=0,因此高斯核值域范围$\\in[0,1]$,特别地，当两个样本完全一样时，值为1，两个样本完全不同时，值为0.\n",
    "\n",
    "\n",
    "有了核函数的概念，我们就动手训练一个核SVM，看看是否能够对线性不可分数据集正确分类：\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "svm = SVC(kernel='rbf', random_state=0, gamma=1., C=10.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SVC(C=10.0, cache_size=200, class_weight=None, coef0=0.0,\n",
       "  decision_function_shape=None, degree=3, gamma=1.0, kernel='rbf',\n",
       "  max_iter=-1, probability=False, random_state=0, shrinking=True,\n",
       "  tol=0.001, verbose=False)"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "svm.fit(X_xor, y_xor)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXYAAAD8CAYAAABjAo9vAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAFMNJREFUeJzt3X2QXQV5x/HvE5LKWxhAEkEghE4YR0cZ1BVH29oKgowK\nKB3RkKk6WKMzto0GtWo6QlU67YgRqc7UWJiqXYOMguBbjSgj1SlC4iAioJMKwfiWBKMEI9UlT/+4\nd8Nms9m9u+fce17u9zOzk5zds+c89+Tmd5/73HPPjcxEktQe86ouQJJULoNdklrGYJekljHYJall\nDHZJahmDXZJaxmCXpJYx2CWpZQx2SWqZ+VXsNOKYhKVV7FqSGmzTjsxcNNNalQR7J9Q3VrNrSWqs\n2NLLWo5iJKllDHZJahmDXZJapqIZ+/6OOuoPXHbZVpYte5R5DXi42bMHNm8+mMsuO4GdOxdUXY4k\n7RVVXI89YiQnv3j64Q/fz+mnL2T+/CcCMfCaZi8ZG3uI22/fxapVJ1ddjKShEJsyc2SmtWrTGy9b\n9miDQh0gmD//iSxb9mjVhWiIJDGnLw2XwqOYiDgYuBV4Qnd7n83MS2e7nc74pWl3wGjE2EjNVGYg\nT7WtwE9Pa6syZuz/B5yRmY9ExALgWxHxlcy8rYRtS0Nj0J31xP0Z8u1SuN/Mjke6iwu6X42+lzzw\nwH1cfPHzeP7zn8CnPnVF1eWo5eowLnFs0y6lnBUTEQcBm4BlwEcz8ztlbLcqRxxxNJdcchXf/Obn\nqy5FLVXnAB2vzS6+uUoJ9sx8DDgtIo4EboiIp2fm3RPXiYiVwMrO0pJC+3vqa57D/J3b9vv+2FGL\nufeTdxTaNsDRRy/m6KMX8+1vf6nwtqTJ6hzqExnwzVXqeeyZ+euIuAU4B7h70s/WAetg/HTHuZu/\ncxtjR+1/HZypwl6qg6aE+VQM+OYpPGOPiEXdTp2IOAQ4C7iv6HalNmjT3Lott2MYlHGy3nHALRFx\nF3AH8LXM/GIJ2x2o6677KBdddBoXXXQa27f/rOpy1AJtDMI23qY2KjyKycy7gGeWUEulLrzwzVx4\n4ZurLkMt0PbwczRTf7W5Vkyd7NjxC1772hF++9uHiZjHtddeyWc+cw+HH35E1aWpxtoe6JMlYbjX\nVCODfeyoxQc8K6YMxxxzLF/60tZStqXhMGyhPs5wr6dGBnsZpzRKZRjWQJ/IcK8fr3QiqTAf4OrF\nYJfmoE2nMZbF41EfBrs0Cwb69Dw29WCwS1LLGOxSD+zUe+dxqp7BPsF733sxZ5+9mFe96ulVl6Ia\nMahmz2NWrcYG++SPai3jo1tf9rLXcdVV/1V8Q5JUoUYG+/XXw+jo42Ge2Vm+/vpi233Ws17AEUcc\nXbxAtYLjl2I8dtVpXLBnwu7dsGHD4+E+OtpZ3r27nM5dMpTK4XGsRuPeeRoBK1Z0/r5hQ+cL4Oyz\nO98P70eShlzjOnbYN9zHGeoqi11muTyeg9fIYB8fv0w0ceYuzZUhpDZoXLBPnKmffTZ84hOdPyfO\n3OdqzZrlXHzx89iy5Ye89KUncOONV5dXuGrPUFdbNHLGfuih+87Ux8cyhx5abBxz+eXryylSjWOo\n95dXgBysxgU7wAUXdDrz8RAfD3dn7JLUwFHMuMkhbqhrruzWB8PjPDi1CfY9e4DGPVXLbt2SVB+F\ngz0iToyIWyLinoj4QUSsmst2Nm8+mLGxh2hOuCdjYw+xefPBVReiAuwiB8vjPRiRBc8RjIjjgOMy\n87sRsRDYBLw8M+858O+MJGzc53tHHfUHLrtsK8uWPcq82jyPOLA9ezoPRpdddgI7dy6ouhzNkUEz\neL6IWkRsysyRmdYq/OJpZv4c+Hn377si4l7geOCAwT6VnTsXsGrVyUXLkXpmqKutSu2NI2Ip8Ezg\nO2VuV1J7+IDaf6UFe0QcDnwOeEtmPjzFz1dGxMaI2Ajby9qtJGmSwjN2gIhYAHwR+Gpmrp15/f1n\n7NKg2TlWy1n7XPQ2Yy/jrJgArgbu7SXUJUn9VcYo5k+AvwLOiIg7u18vKWG7Ut/YrVfPf4P+KeOs\nmG+B/0KSVBcNOGNcUlvZtfeHwS5JLWOwayh5RobazGCXVCnHMeUz2CVVznAvl8EuSS1jsEuqBbv2\n8hjsktQyBruk2rBrL4fBLkktY7BLqhW79uIMdklqGYNdUu3YtRdjsEtSyxjsktQyBrukWnIcM3cG\nuyS1jMEuqbbs2ufGYJekljHYNbTW8YaqS5D6wmDX0NrFQj9HSa1USrBHxDURsS0i7i5je1K/XcEl\nrGatE1y1Ulkd+38A55S0LanvDPXm8AXU2Ssl2DPzVuBXZWxLGoS1rHYMo9Ya2Iw9IlZGxMaI2Ajb\nB7VbaUrrWW64q7UGFuyZuS4zRzJzBBYNarfSlJaznoXs8km+Wml+1QVIVXgraz0lrCHC51Wz5n1b\nQ+lDrPY8drVWWac7rgf+B3hKRGyNiNeXsV2pTKu4kh9zEu/m/axnueexq7Uic/B37YiRhI0D36+G\n2zO4izHmcyi7Wc56T3lsCEcxE8WmzuuU03MUo6HwHZ7DAn7PIfwO8Dz2pjDU58Zg11C4kfPpxEQn\nzj3VUW1msGsofIHzuJDruJU/YznrPY9drebpjhoK53ITq7iSx5jPatYCeB57zTmGmTtfPNVQ+DEn\nsZgdzGeMJ/B7Egz1mjPYp9Lbi6d27BoKT+YX+ywb6vVmqBfjjF1SrRjqxdmxayj8nGPZ0b1G0Qib\nKq5G6i87dkm1YbdeDjt2SZUz0Mtlx66hYGxomNixayjsYDGL2FZ1GZrETr0/DHYNhdO5Y59lP0ez\neoZ6/ziKkTRwhnp/GewaSn7IRjU6l2Ez1PvNUYyG0viHbDiQGQzDfLC8VoyGlnP2/jPQy+a1YiRV\nwDCvnjN2DS0DqFzOz+vDjl3SrBng9VZKxx4R50TEDyNic0S8s4xtama5t0fq31fbGVCzY1feDIU7\n9og4CPgocBawFbgjIm7KzHuKblv7qiJoD7TPJvznPoyHWcoDnMdNfIHzeBK/5IV8nbP4Os+d8Ial\nILmfk1jKg/udKTPVmTO9rFOGQe1nsib822p6ZYxiTgc2Z+aPASLiWuB8wGAvQV275vG+bbrgeYAl\n+3wvge0s3idU++kx5vO/nMLVvIHlfJqLGGUecAzbp1z/3byf3RzKCkYJ4NlsYi2rWcguVvJxoHP+\n+y4Wspq1RPc2TV6nDAfaz+Hs4o0T9jOXsDe426+MYD8e+MmE5a3Ac0vY7tCqa5hPNrnKyctLeXC/\n3zmZB6e8ff0Im2VsZhtP4mGOmDEAE9jNoWzgxQCsYJS3spZvcCbf51TeyLq96z67e6ruatayltWs\nZznLWV9KRz1+FHaxkPUsJ4FLuvu5ir/lVO7iDXyceRz4QWUjz95vu09kO3/MloLVqSkG9uJpRKwE\nVnaWlgxqt43SlEDvhz3E3pAct5z1vI0PTvt7t/OcfS7u9WmW8wiH81sO5xbO4GgeAuBunjHtdnaw\nmFVcyWH8li9wHt/gTADO4Bt8n1P3WXcTI2xihH/n9SxgjHO5iQv4LFu69+vtLN7v2jTj7uekvR/4\nAfAsvrv3Qe1jrOQRFvJW1vIWPsSvOIq380HePuEYPMhSDpriQXD8ged+TtrvZ9fzCrv0IVNGsP8U\nOHHC8gnd7+0jM9dB597XeYOSxg1zoMPjned45zuxE76CS/aOI6byAEv2CcqzuJlRVnALZ3AuN/Fy\nPs8oK9jAixllBSsYnXI7BwriyaE+0X08be86/8Q/9HRbJ/sdh3AIu0ngNxzJdVzIYxzEm/nInKJ4\n+6SrWHaeRSTXc8Gc6lMzlRHsdwCnRMTJdAL91cBFJWy39YY90McFsJBde0M96Iw56H5/uqO0lAf3\nG/ncwCs4l5v4az7OQyzaG+aHsrvyIz45eLdzDMd1P2j7Qq7lYRYyygr+kxX8ksWz3v6BHqA0XEq5\npEBEvAS4EjgIuCYzL59+fS8pYKjvr8yzQO5nyT7XYJ+4relGJVK99XZJAa8VM2AG+mA5W1a79Bbs\nXlJggAz1wfOYaxh5SYEBMWCqM/lMEUcxajuDvc8M9OpNPGsG8LNP1XqOYtR6I2yqugRpoOzY+8RO\nXVJV7Nj7wFCXVCWDvWSGuqSqOYopkaFeXyNs4oEJ13KR2sxgL4GB3gwne3VDDQlHMZLUMgZ7QXbr\nkurGYC/AUJdURwb7HBnqkurKYJ8DQ11SnRnss2SoS6o7g12SWsZgnwW7dUlNYLD3yFCX1BQGuyS1\njMHeA7t1SU1SKNgj4pUR8YOI2BMRM37AqlSl23lO1SVIA1G0Y78buAC4tYRaasluvT38SDwNi0JX\nd8zMewEiDD9Jqgtn7JLUMjN27BFxM3DsFD9ak5k39rqjiFgJrOwsLen11yrlGEZSE80Y7Jn5ojJ2\nlJnrgHUAESNZxjYlSftzFKOhsZQHqy5BGoiipzu+IiK2As8DvhQRXy2nLEnSXBU9K+YG4IaSapEk\nlcBRjCS1jME+jcDXeNvmfk7yHahqPYNdQ2UHi3wHqlrPYJekljHYZ+A4RlLTGOyS1DIGew/s2iU1\nicHeI8O9HY5hO9tYXHUZUl8VeoOS1DQns6XqEqS+s2OfBbv2ZkvgCi6pugyp7yJz8GHVubrjxoHv\nt0xe0reZEpjnA7QaKzZl5owfQ2rHrqHiw7GGgcE+R0E6mmkg/8U0DAz2ggz35khgLaurLkPqO4O9\nBIZ7MwSwkF1VlyH1ncFeEsO9GVby8apLkPrOYC+Rc3dJdWCw94HhLqlKBnufGO715XsQ1HYGex8Z\n7pKqUCjYI+IDEXFfRNwVETdExJFlFdYWhrukQSvasX8NeHpmngr8CHhX8ZLaxxdVJQ1SoWDPzA2Z\nOdZdvA04oXhJkqQiypyxXwx8pcTttY6du6RBmPF67BFxM3DsFD9ak5k3dtdZA4wBo9NsZyWwsrO0\nZA6ltkeQnpkhqW8KX7Y3Il4HvBE4MzN39/Y7zb9sb1EGe7V85qRm6u2yvYU+QSkizgHeAfx5r6Gu\nDrt2Sf1SdMb+EWAh8LWIuDMi/q2EmoaGXaOkfijUsWfmsrIKGVZ27pLK5jtPa8DOXVKZDPaaMNwl\nlcVgl6SWMdhrxK5dUhkM9pox3CUVZbBLUssY7DVk1y6pCINdklrGYK8pu3ZJc2WwS1LLGOw1Ztcu\naS4MdklqGYO95uza+8MLr6nNDHZJahmDvQHs2iXNhsEuSS1jsGtoOWdXWxnsDeE4RlKvDHYNNbt2\ntZHBLkktY7A3iOOY/rBrV9sUCvaIeF9E3BURd0bEhoh4clmFSZLmpmjH/oHMPDUzTwO+CLynhJqk\ngbNrV5sUCvbMfHjC4mHgrKDfHMf0j+GutphfdAMRcTnwGuA3wAunWW8lsLKztKTobiVJBxCZ03eA\nEXEzcOwUP1qTmTdOWO9dwMGZeemMO42RhI2zrVVddpb95bMi1VdsysyRmdaasWPPzBf1uMdR4MvA\njMGuYoI03PsoCcNdjVb0rJhTJiyeD9xXrBxJUlFFZ+z/HBFPAfYAW4A3FS9Jqp5du5qsULBn5l+W\nVYhmx3FM/xnuairfeSpJLWOwS9PwWZGayGBvMMcEg2G4q2kMdklqGYO94ezaJU1msEs9cByjJjHY\nW8CuXdJEBntLGO79Z9eupjDYJallDPYWsWuXBAZ76xju/eU4Rk1gsLeQ4S4NN4O9pQx3aXgZ7C1m\nuEvDyWBvOcNdGj4G+xAw3MvjsVQTGOxDwkCShofBPkSCNOAL8NipKQz2IWTAS+1msEs98IFQTVJK\nsEfEJRGREXFMGdvTYNi598ZjpKYpHOwRcSJwNvBg8XJUBQNeapcyOvYPAe8Ak6HpDPh9eTzUVIWC\nPSLOB36amd/rYd2VEbExIjbC9iK7VZ8Ne6AN++1X882faYWIuBk4doofrQHeTWcMM6PMXAes62xz\nxP81DTAebsN0RUMDXW0QmXO7I0fEM4CvA7u73zoB+Blwemb+Yobf3Q5smdOOH3cMsKPgNqrU5Pqb\nXDs0u/4m1w7Nrr8OtZ+UmYtmWmnOwb7fhiIeAEYycyA3PCI2ZubIIPbVD02uv8m1Q7Prb3Lt0Oz6\nm1S757FLUsvMOGPvVWYuLWtbkqS5a3LHvq7qAgpqcv1Nrh2aXX+Ta4dm19+Y2kubsUuS6qHJHbsk\naQqNDvaIeF9E3BURd0bEhoh4ctU1zUZEfCAi7uvehhsi4siqa+pVRLwyIn4QEXsiohFnCkTEORHx\nw4jYHBHvrLqe2YiIayJiW0TcXXUtsxURJ0bELRFxT/c+s6rqmmYjIg6OiNsj4nvd+v+x6ppm0uhR\nTEQckZkPd//+d8DTMvNNFZfVs4g4G/hGZo5FxL8AZObfV1xWTyLiqcAe4GPA2zJzY8UlTSsiDgJ+\nBJwFbAXuAJZn5j2VFtajiHgB8Ajwycx8etX1zEZEHAccl5nfjYiFwCbg5Q069gEclpmPRMQC4FvA\nqsy8reLSDqjRHft4qHcdRsOuV5OZGzJzrLt4G503eTVCZt6bmT+suo5ZOB3YnJk/zszfA9cC51dc\nU88y81bgV1XXMReZ+fPM/G7377uAe4Hjq62qd9nxSHdxQfer1lnT6GAHiIjLI+InwArgPVXXU8DF\nwFeqLqLFjgd+MmF5Kw0Kl7aIiKXAM4HvVFvJ7ETEQRFxJ7AN+Fpm1rr+2gd7RNwcEXdP8XU+QGau\nycwTgVHgb6qtdn8z1d9dZw0wRuc21EYvtUu9iojDgc8Bb5n0bLv2MvOxzDyNzrPq0yOi1uOw0t6g\n1C+Z+aIeVx0Fvgxc2sdyZm2m+iPidcDLgDOzZi94zOLYN8FPgRMnLJ/Q/Z4GoDub/hwwmpnXV13P\nXGXmryPiFuAcoLYvZNe+Y59ORJwyYfF84L6qapmLiDiHzrXsz8vM3TOtr0LuAE6JiJMj4o+AVwM3\nVVzTUOi++Hg1cG9mrq26ntmKiEXjZ6xFxCF0XoCvddY0/ayYzwFPoXN2xhbgTZnZmC4sIjYDTwAe\n6n7rtqac1RMRrwD+FVgE/Bq4MzNfXG1V04uIlwBXAgcB12Tm5RWX1LOIWA/8BZ0rDP4SuDQzr660\nqB5FxJ8C/w18n87/VYB3Z+aXq6uqdxFxKvAJOvebecB1mfneaquaXqODXZK0v0aPYiRJ+zPYJall\nDHZJahmDXZJaxmCXpJYx2CWpZQx2SWoZg12SWub/ASLP8EPiUHE4AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1149a3550>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_decision_regions(X_xor, y_xor, classifier=svm)\n",
    "plt.legend(loc='upper left')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可以发现核SVM在XOR数据集上表现相当不错.\n",
    "\n",
    "其中参数gamma可以被理解为高斯球面的阶段参数，如果我们增大gamma值，会产生更加柔软的决策界。为了更好地理解gamma参数，我们在Iris数据集上应用RBF核SVM："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "svm = SVC(kernel='rbf', random_state=0, gamma=0.2, C=1.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SVC(C=1.0, cache_size=200, class_weight=None, coef0=0.0,\n",
       "  decision_function_shape=None, degree=3, gamma=0.2, kernel='rbf',\n",
       "  max_iter=-1, probability=False, random_state=0, shrinking=True,\n",
       "  tol=0.001, verbose=False)"
      ]
     },
     "execution_count": 79,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "svm.fit(X_train_std, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAEKCAYAAAASByJ7AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XmYXGWd9vHvnU4gCQFCSISEQIKiwcBAQoIsYZdhE3RA\nmUxYnAgjcomSlkVB3tdBnHFURl4ElAiKCILIOgLCsBgQZc9GWEIA2cKaQAjZyNb9e/+oU0kl6a4+\n3bWcqq77c111VdWpU+f8qiF11/M85zxHEYGZmVmPrAswM7Pa4EAwMzPAgWBmZgkHgpmZAQ4EMzNL\nOBDMzAxwIJiZWcKBYGZmgAPBzMwSPbMuoDP6bdkvBmw3IOsyzMzqytyZc9+LiEEdrVdXgTBguwGc\nOeXMrMswM6srzQOaX0uznruMzMwMcCCYmVnCgWBmZkCdjSG0pVdLL0YsGEG/Vf2yLqUmLem1hDkD\n5rCqaVXWpZhZjav7QBixYATD+g+j3xb9kJR1OTUlIliyYAksgGcGPZN1OWZW4+q+y6jfqn4Og3ZI\not+Afm49mVkqdR8IgMOgCP9tzCytbhEIZmZWOgdCmUy5ZwrjdhrHnjvuyaU/uTTrcszMOs2BUAYt\nLS2ce/q5XH/H9Tw06yFuu+E25jw3J+uyzMw6pe6PMuqMAXseSo/572+wvHXQlix47J4ub3fGEzPY\n/hPbM+zjwwD4p/H/xD133MOIkSO6vE0zs2prqEDoMf99WgduODleWyHRGW+/9TZDhg5Z83zwNoOZ\n/sT0krZpZlZt7jIyMzPAgVAWg4cM5q033lrz/O0332bwNoMzrMjMrPMcCGUwavdRvPzSy7z2ymus\nXLmS//nD/3DIkYdkXZaZWac01BhCpfTs2ZMf/uyHTPjcBFpaWpgwcQI77rRj1mWZmXVKQwVC66At\n2z3KqFQHH34wBx9+cMnbMTPLSkMFQimHlpqZdXceQzAzM8CBYGZmCQeCmZkBDgQzM0tkFgiStpX0\ngKTnJD0raVJWtZiZWbYthNXAmRExEtgTOE3SyAzr6bLmf2tmpyE7sf+o/bMuxcysyzILhIh4OyKm\nJ48XA7OBbSq/3+LPu2L8v47n93f+vvQNmZllqCbGECQNB0YDj7fx2imSpkqauuS9JSXt59or+zD5\n4r5rQiACJl/cl2uv7FPSdvfady/6D+hf0jbMzLKWeSBI6gfcAjRHxKL1X4+IKyJibESM7Tew6xeL\nj4AlS8RtN/ReEwqTL+7LbTf0ZskSlaWlYGZWzzI9U1lSL3JhcF1E3FrZfcGpzcsAuO2G3tx2Q28A\njv6X5ZzavAxfi97MGl2WRxkJ+DUwOyIuqs4+14ZCnsPAzCwnyxbCOOBE4GlJM5Nl342Iu7q6wbkz\ntyr6egRcdx2s+mjtsh+dvRXHH0+HobDtqHe7WpaZWV3ILBAi4m9Ap3+bNw9Y93SFu++eTdNHxYMg\nt79cGNx7LxxyCBx//NrnQIehUCxszjtvAjNnTWHBewsYPXw0Z3/vbI476bhUn8fMrFbU1Wync2d+\nrMvvlaBv37VhIOXuIbe8lG6j//zPDQ85nTtz7WO3LsysHtRVIJTqmGNyLYX8l38+FCo9hrB+68IB\nYWa1qKECATb88s9iQLmt7ieHhJllreECoVa5FWFmWXMg1Ci3Isys2hwIdcStCDOrJAdCHSsMCIeD\nmZXKgVAG77wzl/PP/zILFrwLiKOPPoUJE6p7eQeHg5mVyoFQBj179qS5+afsuONuLF26mC9/eQx7\n7PGPfPzj2VzeweFgZl3RcIGwoM+rvNV/Gst7LaL3qs0YsnAMAz4aXtI2Bw4czMCBgwHYZJNNGT78\n08yf/2ZmgVBo7syt+OC9BTTvnGuxXLzgZxlXZGa1qqECYUGfV3l50AOotYmmlo1Z0bSUlwc9APMP\nLDkU8t5661XmzJnBTjvtUZbtldv6U384IMwsr91AkHRMivcvL2Uyump7q/+0XBhELwCaohctrbnl\n5QiEZcuW8J3vfJEzzriYfv02K3l71ZAPCAeDmRVrIVwJ/JHiE9DtB9RNICzvtYimlo3XWdYjerK8\n1wbX5em01atX8Z3vfJHDDjuegw5Kk6W1pbDl4HAwa0zFAuHuiDip2Jsl/a7M9VRU71WbsaJp6ZoW\nAkCrVtN7VWm/5iOCH/zgZIYP/zTHH39GqWVmzuFg1pjaDYSIOKGjN6dZp5YMWTiGlwc9QEtrrmXQ\nqtVEjxaGvD+mpO0+9dTD3HXXteywwz9w3HGjADjttB8ybtwR5Sg7Uw4Hs8bR5TGESl/yshIGfDQc\n5h+47lFG75d+lNGoUfvw5JPd/6LMHm8w696KdRkdldx/DNgbmJI8PxB4BKi7QIBcKJTriKJG5VaD\nWfdUrMvoKwCS7gVGRsTbyfPBwNVVqc5qnsPBrPtIcx7CtvkwSLwLbFeheooawzSmIkT3756pRw4H\ns/qWJhD+LOkeIH+dyPHA/ZUrqWPhUKh5Hm8wqz89OlohIr4BTAZ2TW5XRMQ3K11YRwIRRU+RsFrQ\nPGDSBmdHm1ltSjt1xXRgcUTcL6mvpE0jYnElC0trGK+xlE9nXYZ1wN1JZrWvwxaCpK8CNwO/TBZt\nA/xPJYvqrLFMZSxTM9n34sULuemmX3T5/ddffzHLly8ruY5p0x7kqaceKXk71eBWg1lt6jAQgNOA\nccAigIh4kdyhqDUni1BYvHghN9/c9UC44YbyBcKsWfURCHn5YHA4mNWGNF1GKyJipZTrr5fUE2p3\nRDcfClMZW5X9XXbZObz55t857rhR7LHHPzJp0oVce+2F3HffjaxatYIDDjiar33t+3z00VLOPfef\nmTfvDVpaWjj55P/LggXvMn/+W5x66oH07z+QyZMfWGfbl156Dn/96+00NfVkjz0Oobn5v/ngg/n8\n13+dyjvvvA7AmWdezKBB23DLLZNpamri7rt/x9lnX8ro0ftW5fOXi7uUzLKXJhD+Ium7QB9J/wh8\nHbijsmXVj29840f8/e/PcP31MwF47LF7ef31F/ntb58gIjjzzM8zffpDLFw4n4EDh3DxxX8CYMmS\nD+nXb3Ouv/4iJk9+gP79B66z3YUL3+fBB2/j5pufRxKLFy8E4Kc/ncRxx32LUaP24Z13Xueb3zyU\nm26azRe/eCp9+vTjxBPPqu4foAKaB0xyKJhlIE0gnAOcDDwNfA24KyKurGhVZVDtlkLeY4/dy+OP\n38vxx48G4KOPljB37ouMGrUvF198Jpde+h322efIDn/B9+u3ORtv3Jsf/OBk9tnnSPbd90gAnnji\nfl5++bk16y1duohly5ZU7gNlxC0Gs+pLEwjHAzcUhoCkIyPizsqVVT5jmVrVUIgIJk48l2OO+doG\nr1177XQefvguLr/8/7D77p/lq1/9Xrvb6dmzJ1df/QRPPvln/vznm7nppsu4/PIptLa28pvfPMbG\nG/eu5MeoKQ4Hs+pIM6h8KfBXSYXHdl5QoXoqopJHIfXtuynLlq09AnevvQ7l9tuvWvOrfd68N1mw\nYB7z579F7959OeKIEzjxxLOZM2f6mvcvXbrhEbzLli1hyZIPGTfuCM444//x4otPAbDnnodw442X\nrllvzpyZbdbRXXkg2qxy0rQQXiHXZXSzpPMj4iaKXzSnZlWitdC//5bsuus4xo/fmb33PpxJky7k\nlVdmc9JJewHQt28/Lrjgd8yd+xKXXHI2Ug969uzFOedcDsDRR5/C6acfxqBBQ9YZVF62bDFnnvkF\nVq5cTkTQ3HwRAGeddQk//vFpTJiwCy0tqxk9ej/OPXcy++57FOec8yX+8pc/1uWgclf4bGiz8lJE\n8QOGJE2PiN0kDSQ3fcVTwCERsUs1Ciw0Vor1f+fPvvtuPj1wYJvrt6fa4wpZe++92Rx+ePc/ec/B\nYNa25gHN0yKiwy++NC2EtwEi4j1JhwI/BnYusb5MVXtcwarDYw1mpUkzl9HnCh63RsTZEZFm7KGm\nZXVms1WHxxjMOq/YFdMujohmSXfQxoloEfH5ilaWVmsrQdcGNbI6NLW6gtbWrGvIhlsMZp1TrMvo\n2uT+vyu1c0lXAUcC8yKiS91QvV96ifcHDGDLnj27PNLdfbuQgtWr3+ellxrnENX2OBzMOtbhoHJF\ndy7tBywBrkkTCG0NKq/aYgveOP98lu+wA/QovSfrNYaVvI1a0doKL73Um/PPH8oHH/TKupya4lCw\nRpJ2ULndQJD0NEXmLCrXUUaShgN3djUQKsEX32ksDgfr7spxlNGRyf1pyX2+C+kEqji5naRTgFOg\netftzF94x8HQGDx3kllOmvMQZkTE6PWWTY+I3cpSQA22EPIcCI3J4WDdTdoWQppOd0kaV/Bk75Tv\nq3u+RGdj8iGr1qjSnJh2EvAbSZsnzxcmyxqCu48ak49KskZUNBAk9QB2iIhd84EQER+Wa+eSfg8c\nAAyU9Abw7xHx63Jtv5wCORQalMPBGkXRrp+IaAW+nTz+sJxhkGxzQkQMjoheETG0VsMgL5C7kRqc\nu5OsO0vTZXS/pLOAPwBL8wsjYkHFqqpxbi00NrcYrLtKEwjjk/vTCpYF8PHyl1M/HAoG2U7BHQFS\n+8/NOqvDQIiI7atRSD1yKFhetYPhkat3ZusR89l+z3eRcmHQulqoKcpxwr41qDQtBCTtDIwE1kyK\nExHXVKqoeuJQsELVCIYI2HrEfIbtPo/WFujRlFveo2cQrW4pWNd1GAiS/p3ckUAjgbuAw4G/AQ6E\nhA9NtfVVcpxBgu33fJfWFmhd3USPHi1rpvtVD4eBdV2axuWXgM8C70TEV4Bdgc2Lv6Ux+Qgka0sl\njkyS1rYMCv+3cxhYKdJ0GX0UEa2SVkvaDJgHbFvhuuqWWwvWlnK3GCKgtSX/hDWh4O4iK0WaFsJU\nSf2BK4FpwHTg0YpW1Q24tWDtKbXFEAGvPLYVIHr0bFmnhZAfQ1h/irIMZ7m3OpLmKKOvJw8nS/pf\nYLOImFXZsroHDzhbe0ppMUjwzpxBAG0eZbR6ZQ/mTh+0zmsL3+hHn/4r6L3pqrJ+Duteil0Poehs\nphExvSIVFVHt2U7LxaFgaXQ2GNo6DwHgwZ+PZvqtI9jtmDkccNqMDZ67S6nxlON6CD9N7nsDY4Gn\nyDVOdwGmAnuVWmSjcEvB0njhoaFs9rGlbL3jB6nWX/+LPf/8gNNmADD91hFMv3UEgMPAUml3DCEi\nDoyIA4G3gd0iYmxEjAFGA29Wq8DuwvMgWUe2HLaIlR/15PUZH+P1GR/r8naktaGQ5zCwNNIMKo+I\niKfzTyLiGeDTlSupe3MoWHvef20zNuqzes3z12d8jHee34IXHhoKpB8ojsh1G63/PL++B5itPWkC\nYZakX0k6ILldCXhQuQQOBWvLlEvG0HPjFjbqswoINuqzip4btzDlkjE0D5i0wZf6gz8fzSNXr3uh\nwfzy/JjBnl9+hs23XsK0W0bw4M9H09ra9vvMIF0gfAV4FpiU3J5LllkJHAq2vuenDOfmsw9k0bxN\n2GTAChbN24Sbzz6Q56cMB+CP39uPb205aZ0v/RVLeq3zi1+CjfutYrdj5rD/12ewckkvPnynH5tv\nvYSNNlnFX37R9vvMIMU1lWtJvR5l1BEPOFtXfOGCh9odG8gfgZQPj2m3jFizngeYG0/ZrqksaZyk\n+yS9IOnl/K08ZRq4tWBdk28xtCX/ZZ8fYC788ncYWHvSTF3xa+Bb5M5SbulgXesiT3lhXVXsJLf1\nB5gh99yhYG1JM4bwYUTcHRHzIuL9/K3ilZlZp+XDobV1wwHmM/58A7sdM4fpt45YZ4DaLK/DMQRJ\nPwKagFuBFfnlPlO5ctxKsFId+u1HGfVPL7LFNktYtbwH/QauWDOmMOuOj7N0QV/2nvhM1mValZTj\nTOW8PZL7wo0FcFBXCrOOufvISrHjQa+y+/jnWb2iiUXv9mHQJxatM6awy1Evu7vI2pRmcrsDq1GI\nbchTXlhXHHT6NFavaGLlR70YtMMSVq1oQgG9+uSGAB0G1p60l9D8HLAT615C84JKFWVrubVgnbXl\nsEUsXbAxAL02aqGlJTdtyuoVPXjruYFsN3pexhVarUpz2OlkYDzwTXKT2x0LDKtwXbYeH5pqaRVO\ngbFqZRPqAT0UrF6Zu8Ra4XQYZoXSHGW0d0R8GfggIr5PbpbTT1W2LDPrqsIpMBa/24cePQI1tbJo\nXp8NpsMwK5TqEprJ/TJJQ4D3gcGVK8na4+4jSyM/BcZBp09jy2GLeGfOFgjRe7OVLHxzE6ZcMmbN\ndBjlvrSn1bc0gXBncgnNC8ldPjOAX1W0KivKg83WkeenDF/zpZ9WPhwcDI0rzXkIG0fEivxjcgPL\ny/PLqqlRzkPoDAeDVYqDofso21xGwKP5BxGxIiI+LFxm2fJgs1VK84BJHmdoMO0GgqStJY0B+kga\nLWm35HYA0LdqFVqHHApWSQ6FxlFsDOFQYCIwlNz1lfPfOouB71a2LOssjytYJTUPmMSOB73KQadP\n41P7vZF1OVYhacYQvhgRt1SpnqI8hpCOg8GqwWMM9aOcYwhDJW2mnF9Jmi7pkDLUaBXiLiSrBncl\ndT9pAuGkiFgEHAJsCZwI/KgcO5d0mKQ5kl6SdE45tmk5DgWrhvzAs8Ohe0gTCPlvliOAayLi2YJl\nXSapCfg5cDgwEpggaWSp27W1HApWTQ6F+pcmEKZJupdcINwjaVOgtQz7/gzwUkS8HBErgRuAL5Rh\nu1bAoWDV5NZCfUsTCCcD5wC7R8QyYCPgK2XY9zbA3ILnbyTLrMwCORisqhwK9andw04lbR0R70RE\nK7kpKwBILp/5fuE6lSxQ0inAKQDbVXJHDcCHplo1eZ6k+lOshXBXivenWac9bwLbFjwfmixbR0Rc\nERFjI2LsoBJ2ZjluLVgW3JVUH4oFwq6SFhW5LQa2KmHfTwKflLS9pI2AfwFuL2F7ZlbjHAq1rd0u\no4hoquSOI2K1pG8A9wBNwFXJEUxWBe4+sqy4K6l2pRlUrpiIuCsiPhURn4iI/8yylkbk7iPLmlsM\ntSXTQLDa4FCwLHl8oXY4EAxwa8Gy51DIXporpuXPKt6qcP2IeL1SRZlZY/L4QrY6bCFI+ibwLnAf\n8KfkdmeF67KMuJVgtcIthupL00KYBIxITkizBpAPBR+FZFnzdZ6rK80Ywlzgw0oXYrXHrQWrFW4t\nVEexqSvOSB6+DDwo6U/AivzrEXFRhWuzGuDWgtUKtxYqr1iX0abJ/evJbaPkBvjbodH4RDarFR54\nrpxiZyp/H0DSsRFxU+Frko6tdGFWexwKVmuaB0xyKJRRmjGEc1MuswbgcQWrNT6xrXyKjSEcTu6i\nONtIuqTgpc2A1ZUuzGqXxxWsFrm1ULpiYwhvAdOAzyf3eYuBb1WyKKsP7kKyWuOB59IUG0N4CnhK\n0nURsaqKNZmZlcStha4p1mX0NMnRRNKG/cYRsUvlyrJ64e4jq1U+GqnzinUZHZncn5bcX5vcn4AP\nO7X1uPvIaplbDOkoovg/YkkzImL0esumR8RuFa2sDWOlmFrtnVqnORisljViMDQPaJ4WEWM7Wi/N\nYaeSNK7gyd4p32cNyoemWi3zIartSzO53cnAVZI2BwR8AJxU0arMzCrIRyO1rcNf+hExLSJ2BXYF\ndomIURExvfKlWT3zBXesHri1sK5iRxmdEBG/K5jkLr8c8OR2lo4Hm63WubWwVrEWwibJ/abt3MxS\ncWvB6oFbC8VPTPtl8vDHEbG8SvVYN+bWgtW6Rm8tpBlUfkbSu8Bfk9vfIsIXzLEucShYPWjUYEgz\nqLwDMAF4GvgcueksZla6MOu+3H1k9aLRZlLtMBAkDQXGAfsCo4FngT9UuC7r5hwKVk8aJRTSdBm9\nDjwJ/DAiTq1wPdZAPA+S1ZNG6EZKc8bxaOAa4DhJj0q6RtLJFa7LzKwmdefWQocthIh4StLfgb+T\n6zY6Adgf+HWFa7MG4ZaC1Zvu2lpIM4YwFXgUOBqYDewXEcMqXZg1Ho8rWL3pbq2FNGMIh0fE/IpX\nYoZbC1Z/ulNrIc1hpw4Dqzq3FqzedIfWgqextprlULB6U+/nLTgQrKY5FKwe1WsoFJvt9Jhib4yI\nW8tfjtmGPN2F1aN6HFsoNqh8VJHXAuhyIEg6Fjgf+DTwmQhfGdOK82CzWeUVm+30KxXc7zPAMcAv\nO1rRrJBbC1ZvmgdMqptWQprDTpH0OWAnoHd+WURc0NWdRsTsZLtd3YSZWd2ol+6jNCemTQbGA98k\nd03lYwGfmGaZ8QV3rF7V+mBzmhbC3hGxi6RZEfF9ST8F7u7oTZLuB7Zu46XzIuKPaQuUdApwCsB2\nad9kDcHdR1aParm1kCYQPkrul0kaArwPDO7oTRFxcCmFFWznCuAKgLGS//XbOjzYbPWqFscW0gTC\nnZL6AxcC08kdYfSrilZlZtYAaq21kObEtJ9ExMKIuIXc2MGOwH+UslNJR0t6A9gL+JOke0rZnpnH\nFaye1crYQppAeDT/ICJWJNdTfrTI+h2KiNsiYmhEbBwRW0XEoaVszyzPoWDWdcXOVN4a2AboI2k0\nrPmXthnQtwq1mXWJxxWsHtVC91GxMYRDgYnAUOCiguWLgO9WsCazsvBRSFaPshxsLnam8m+B30r6\nYjJ+YGZmVZBVayHNGMLDkn4t6W4ASSN9TWWrFx5sNksvTSD8BrgHGJI8fwForlhFZhXgULB6VO3r\nK6QJhIERcSPQChARq4GWilZlVgFuLVi9qlYopAmEpZK2JHdCGpL2BD6saFVmZraOaoRCmkA4A7gd\n+ISkh4FryE10Z1aX3EqwelXpUOgwECJiOrA/sDfwNWCniJhV0arMKszdR1avKhkKHc5lJKk38HVg\nH3LdRn+VNDkillesKrMq8bkKZmulmdzuGmAxcGny/DjgWnLXRTCrew4FqzeVOk8hzRjCzhFxckQ8\nkNy+Su7qaWbdhruPrB6Vu/soTSBMT44sAkDSHsDUslZhVgM8rmD1qJyhkCYQxgCPSHpV0qvkZjrd\nXdLTkjy4bN2OQ8HqTblCIc0YwmFl2ZNZHfG4gjWiDgMhIl6rRiFmtcahYPWkHAPNabqMzBqWu4+s\nkTgQzDrgwWarJ6WMJzgQzFJyKFi96GooOBDMOsGtBasXXQkFB4KZmQEOBLMucSvB6kFnWwmKqJ/D\n6sZK4VOkrdb40FSrfZoWEWM7WsstBLMSubVg3YUDwawMHArWHTgQzMrEoWD1zoFgVkYOBatnDgSz\nMvO5ClavHAhmFeJQsHrjQDCrIIeC1RMHglmFORSsXjgQzKrA4wpWDxwIZlXkULBa5kAwMzMgo0CQ\ndKGk5yXNknSbpP5Z1GGWBXcfWa3KqoVwH7BzROwCvACcm1EdZplxKFitySQQIuLeiFidPH0MGJpF\nHWZZcyhYLamFMYSTgLuzLsIsKw4FqxU9K7VhSfcDW7fx0nkR8cdknfOA1cB1RbZzCnAKwHYVqNOs\nFgTydRUsc5ldIEfSROBrwGcjYlma9/gCOdYIHAxWfukukFOxFkIxkg4Dvg3snzYMzBqFWwuWlazG\nEC4DNgXukzRT0uSM6jCrSR5XsCxk0kKIiB2y2K9ZPXFLwaqtFo4yMrN2+CQ2qyYHglkdcChYNTgQ\nzMwMcCCY1Q13H1mlORDM6oxDwSrFgWBWh9xasEpwIJiZGeBAMKtrbiVYOTkQzOqcu4+sXBwIZt2E\nQ8FK5UAw60YcClYKB4JZN+NQsK5yIJh1Qx5XsK5wIJh1Yw4F64zMrpjWFZLmA69lXUeBgcB7WReR\noUb+/I382aGxP389fvZhETGoo5XqKhBqjaSpaS5L11018udv5M8Ojf35u/Nnd5eRmZkBDgQzM0s4\nEEpzRdYFZKyRP38jf3Zo7M/fbT+7xxDMzAxwC8HMzBIOhBJJulDS85JmSbpNUv+sa6omScdKelZS\nq6RueeTF+iQdJmmOpJcknZN1PdUk6SpJ8yQ9k3Ut1SZpW0kPSHou+X9+UtY1lZsDoXT3ATtHxC7A\nC8C5GddTbc8AxwAPZV1INUhqAn4OHA6MBCZIGpltVVV1NXBY1kVkZDVwZkSMBPYETutu/+0dCCWK\niHsjYnXy9DFgaJb1VFtEzI6IOVnXUUWfAV6KiJcjYiVwA/CFjGuqmoh4CFiQdR1ZiIi3I2J68ngx\nMBvYJtuqysuBUF4nAXdnXYRV1DbA3ILnb9DNvhSsY5KGA6OBx7OtpLx6Zl1APZB0P7B1Gy+dFxF/\nTNY5j1yT8rpq1lYNaT6/WaOQ1A+4BWiOiEVZ11NODoQUIuLgYq9LmggcCXw2uuFxvB19/gbzJrBt\nwfOhyTJrAJJ6kQuD6yLi1qzrKTd3GZVI0mHAt4HPR8SyrOuxinsS+KSk7SVtBPwLcHvGNVkVSBLw\na2B2RFyUdT2V4EAo3WXApsB9kmZKmpx1QdUk6WhJbwB7AX+SdE/WNVVScgDBN4B7yA0q3hgRz2Zb\nVfVI+j3wKDBC0huSTs66pioaB5wIHJT8W58p6Yisiyonn6lsZmaAWwhmZpZwIJiZGeBAMDOzhAPB\nzMwAB4KZmSUcCFY1kiZKGpJivaslfSnt8jLU9d2Cx8PTzOSZ1PKKpFOLrDOqnIclJn+/y0rcxoP5\nWWkl3VXq7LySDpB0Z/J4fDID7J2lbNOy40CwapoIdBgIGfhux6u06eyIKHbeySggs+PUJRWdiSAi\njoiIheXaX0T8Afi3cm3Pqs+BYF2S/JJ+XtJ1kmZLullS3+S1MZL+ImmapHskDU5+2Y8FrktO6Okj\n6XuSnpT0jKQrkjNB0+5/g30kyx+U9GNJT0h6QdK+yfK+km5M5rK/TdLjksZK+hHQJ6kpPw9Vk6Qr\nkznv75XUJ0U9xyaf4ylJDyVnMV8AjE+2PV7SZyQ9KmmGpEckjUjeO1HSrZL+V9KLkn5SsN2vJJ/j\nCXInRuWXH5V8hhmS7pe0VbL8fEnXSnoYuDb5O9+Q/De6DehTsI1XJQ2UdGrBiVavSHogef2QpN7p\nkm5Sbg6f/PUgnpc0ndzU59ZdRIRvvnX6BgwHAhiXPL8KOAvoBTwCDEqWjweuSh4/CIwt2MaAgsfX\nAkclj69XvDMjAAADoElEQVQGvtTGPq8GvpRiHz9NHh8B3J88Pgv4ZfJ4Z3ITEY5Nni9Z73OtBkYl\nz28ETmivloLnTwPbJI/7J/cTgcsK1tkM6Jk8Phi4pWC9l4HNgd7Aa+TmSxoMvA4MAjYCHs5vD9iC\ntSeW/lvBZz4fmAb0SZ6fUfC32WW9z/0qMLCgvl7AX4GjgIHkrnGxSfLad4DvJfXNBT4JKPn73Fmw\njQMKn/tWXzdPbmelmBsRDyePfwecDvwvuS/c+5If/E3A2+28/0BJ3wb6AgOAZ4E7Uux3RAf7yE86\nNo3cFzzAPsDPACLiGUmzimz/lYiY2cY2inkYuFrSjQX7X9/mwG8lfZJcmPYqeO3PEfEhgKTngGHk\nvpQfjIj5yfI/AJ9K1h8K/CFpGW0EvFKwrdsj4qPk8X7AJQARMauDz/0zYEpE3CHpSHIXAHo4+Rtv\nRG7Kih3J/X1eTGr6HXBKkW1aHXEgWCnWn/ckyP1qfDYi9ir2Rkm9gV+Q+7U6V9L55H59ptHRPlYk\n9y107f/xFQWPWyjoZmlPRJwqaQ/gc8A0SWPaWO0HwAMRcbRy8+k/WGSfHdV9KXBRRNwu6QByLYO8\npR3Vuz7lZuwdRm6eJsj9je+LiAnrrTeqs9u2+uExBCvFdpLyX8rHAX8D5gCD8ssl9ZK0U7LOYnIT\nAcLaL//3kr7pzhw9VGwf7XkY+Odk/ZHAPxS8tkq5aY27TNInIuLxiPgeMJ9cl0/h54VcCyE/VfbE\nFJt9HNhf0pZJfce2s61/LbKNh8j9t0HSzuS6jdavfQy5LrUTIqI1WfwYME7SDsk6m0j6FPA8MFzS\nJ5L1Jqy/PatfDgQrxRxy15WdTa5P+/LIXVbyS8CPJT0FzAT2Tta/GpgsaSa5X8RXkrsm8z3kppVO\npYN9tOcX5ELkOeA/yHVPfZi8dgUwq2BQuSsulPS0coesPgI8BTwAjMwPKgM/Af5L0gxStFwi4m1y\nv/wfJRdoswtePh+4SdI04L0im7kc6Jf8N7qAXBfY+r5BrsvugaTWXyXdVBOB3yfdTI8CO0bEcnJd\nRH9KBpXndfQ5rH54tlPrkqTL486I2DnjUlKR1AT0iojlya/b+4ERSbh0ZXtXk/v8N5exzLqXdF+d\nFRFHZl2LdZ7HEKxR9CX3C7gXuf7xr3c1DBIfAj+QNDCKn4vQMJJW0L/TdivE6oBbCGZmBngMwczM\nEg4EMzMDHAhmZpZwIJiZGeBAMDOzhAPBzMwA+P+ZeZUZYiuFbAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x11497cbd0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_decision_regions(X_combined_std, y_combined, classifier=svm, test_idx=range(105, 150))\n",
    "plt.xlabel('petal length [standardized]')\n",
    "plt.ylabel('petal width [standardized]')\n",
    "plt.legend(loc='upper left')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "我们选择的gamma值相对比较小，所以决策界比较soft，现在我们增大gamma值，然后观察决策界："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "svm = SVC(kernel='rbf', random_state=0, gamma=100, C=1.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SVC(C=1.0, cache_size=200, class_weight=None, coef0=0.0,\n",
       "  decision_function_shape=None, degree=3, gamma=100, kernel='rbf',\n",
       "  max_iter=-1, probability=False, random_state=0, shrinking=True,\n",
       "  tol=0.001, verbose=False)"
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "svm.fit(X_train_std, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAEKCAYAAAASByJ7AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XmYFOXZ7/HvPTACMqCMEEQRMImBIAcZJYq7LIoGTSLR\nKIkxuITjUaOoeY3LeQ2S5UQ9GoiSEFyCUaMxIlFRgyIiCq4sKiJoFAUEESQCg4gDc79/dPXQDDM9\nNd1dXdPTv8919dVdSz9114h1dz1bmbsjIiJSEncAIiLSNCghiIgIoIQgIiIBJQQREQGUEEREJKCE\nICIigBKCiIgElBBERARQQhARkUDLuANojLK9yry8W3ncYYiIFJQVC1esc/dODe1XUAmhvFs5V8y8\nIu4wREQKyujy0R+G2U9VRiIiAighiIhIQAlBRESAAmtDqEvp9lJ6ru9JWVVZ3KE0SZWllSwtX0pV\ni6q4QxGRJq7gE0LP9T3pvmd3yjqUYWZxh9OkuDuV6ythPSzqtCjucESkiSv4KqOyqjIlg3qYGWXl\nZbp7EpFQCj4hAEoGaehvIyJhNYuEICIi2VNCyJGZ02dy5IFHMqDXAG698da4wxERaTQlhBzYvn07\nV19yNX977G/MfmM2Ux+YytLFS+MOS0SkUQq+l1FjlA8YSsnaT3dZX91pL9a/ND3jche8soD9v7Y/\n3b/aHYDvnfE9pj82nZ69e2ZcpohIvhVVQihZ+ynVHXedHK+uJNEYq1etZp+u+9Qsd9m3C/NfmZ9V\nmSIi+aYqIxERAZQQcqLLPl1YtXJVzfLqj1bTZd8uMUYkItJ4Sgg50O9b/Xj/3+/z4bIP+fLLL/nn\n3//JCSefEHdYIiKNUlRtCFFp2bIlvx3/W0YMG8H27dsZMXIEvQ7sFXdYIiKNUlQJobrTXvX2MsrW\nkJOGMOSkIVmXIyISl6JKCNl0LRURae7UhiAiIoASgoiIBJQQREQEUEIQEZFAbAnBzPYzs2fNbLGZ\nvWVml8YVi4iIxHuHsA24wt17AwOAi8ysd4zxZGz0+aM5cJ8DObbfsXGHIiKSsdgSgruvdvf5wedN\nwNvAvtEfN/1yJs74yRncP+3+7AsSEYlRk2hDMLMeQAXwch3bRpnZa2b2WuW6yqyOc8/tbZg4bvea\nJOAOE8ftzj23t8mq3MOPPpw9y/fMqgwRkbjFnhDMrAyYAox29421t7v7JHfv7+79yzpm/rB4d6is\nNKY+0LomKUwctztTH2hNZaXl5E5BRKSQxTpS2cxKSSSD+9z94WiPBReM/hyAqQ+0ZuoDrQE49cwv\nuGD05+hZ9CJS7OLsZWTAncDb7n5Lfo65IykkKRmIiCTEWWV0JPBjYJCZLQxe347ygMlqolSpbQoi\nIsUszl5GL7i7uXtfd+8XvJ6I7ng72gxOPfMLpr+0nlPP/GKnNoVMXXDWBZx89Mm8t/Q9KnpU8Le7\n/pa7wEVE8qRoZjs1g7Iy36nNIFl9VFbmWVUbTbx3Yo6iFBGJT9EkBIAf/3QL7tRc/JNJQW0IIiJN\noNtpvtW++CsZiIgkFF1CEBGRuikhiIgIoIQgIiIBJQQREQGUEHLioxUfMXzIcI7uezTHHHQMt//h\n9rhDEhFptKLqdhqVli1bMubGMfQ9uC+Vmyo54bATOGbIMfTs3TPu0EREQiu6hLCqahVLqpawuXoz\nbUva0qu0F/uU7pNVmZ27dKZzl84AlLUr44BeB/Dxqo+VEESkoBRVQlhVtYp5W+dRQgmllLKlegvz\nts4DyDopJC3/YDmLFi7i4EMPzkl5IiL5Um9CMLPhIb7/RZTzD+XakqollFBCS0ucdktass23saRq\nSU4SwubKzZz/g/MZe/NY2rVvl3V5IiL5lO4O4XbgESDdWN5jgIJJCJurN1NK6U7rWtCCzdWbsy67\nqqqK835wHsNHDGfYqcOyLk9EJN/SJYQn3f3cdF82s3tzHE+k2pa0ZUv1FlqmnPZ2ttO2pG1W5bo7\nl/30Mg7odQAXXHZBtmGKiMSi3m6n7n5WQ18Os09T0qu0F9VUs8234e5s821UU02v0l5ZlfvKnFd4\n6L6HeOHZFxh8yGAGHzKYGU/OyFHUIiL5kXEbQtSPvIxCsp0g172MDjvqMD6u+jgXIYqIxCZdldEp\nwftXgCOAmcHyQGAuUHAJARJJIVc9ikREmpN6E4K7nwNgZk8Bvd19dbDcBZicl+hERCRvwkxdsV8y\nGQTWAN0iikdERGISZmDaM2Y2Hbg/WD4DUIupiEgz02BCcPeLzexUEmMOACa5+9RowxIRkXwLO3XF\nfGCTu88ws93NrJ27b4oyMBERya8G2xDM7KfAQ8Cfg1X7Av+MMqhCsuGzDfzlT3/J+PuTxk/i888/\nzzqOOc/N4dW5r2ZdjogUrzCNyhcBRwIbAdz9XRJdUYVEQpj858kZf3/SrZPY8vmWrOOY+9xcXn1J\nCUFEMhcmIWx19y+TC2bWEvDoQiosv7n2N3z43ocMPmQw1//iegAm3DyBoQOGMrBiIDdefyMAmzdv\n5kff+RGDDh7Esf2O5Z8P/pM7br2DNavW8P0h32f4kF3HAf76ml9zdN+jGVgxkDFXjgFg3dp1nPeD\n8xg6YChDBwzllTmvsPyD5fx10l+ZNH4Sgw8ZzEsvvJS38xeR5iNMG8JzZnYN0MbMjgcuBB6LNqzC\nce1vrmXJW0t4Zt4zAMx6ehbL3l3Gv178F+7O2aeezYvPv8inaz9l7y57c9+j9wGwccNG2u/Rnonj\nJzJlxhT26rjXTuWu/3Q9Tz7yJC8segEzY8NnGwD478v/m1GXjOKwow5j5fKVjBg2gufffJ6zR51N\n27K2XHj5hfn9A4hIsxEmIVwFnAe8Cfxv4Al31zMi6zHr6VnMmjGLIf2HAIk7g2XvLuOwow5jzJVj\n+NXVv+L4Yccz4KgBactpv0d7WrVqxWU/vYzjhx3P8cOOB2D2M7N5Z/E7Nftt2riJzZXZz9YqIhIm\nIfwIeCA1CZjZye4+LbqwCpe7c8mVl3D2qLN32fb0K0/zzJPP8LvrfsfRg47miv97Rb3ltGzZkidf\nfJLnZz7PtCnTuOuPdzHl6Sl4tfP4nMdp3bp1lKchIkUoTBvCrcDzZvbNlHVjI4qn4JS1K6NyU2XN\n8sATBnL/5PtrfrWv/mg1az9Zy8erPqbN7m047UenceEVF/LmgjcT3y/b+ftJmys3s3HDRoacNISx\nN49l8RuLATj2+GO5c8KdNfstWriozjhERBorzB3CMhJVRg+Z2Rh3/wfpH5pTVMr3KufQIw7l2H7H\nMmjoIH55wy959+13GXZU4iE5bcvaMuHuCSx7bxljfzGWkpISWpa25IbbbgDgrPPP4ocn/5DOXTrz\n8Iwd8wVWbqrkJ8N/wtYvtuLujLlpDAC//v2vufqSqxlYMZBt27dx+FGHc+Mfb+SEYSdw/hnnM/3R\n6fxm/G8arJISEanN3NN3GDKz+e5+sJl1JDF9xevACe7eNx8BpupW0c2vmLlzNcuAVQPo0bNHvkMp\nKB8s/YCX9lHPI5FiNbp89Dx379/QfmGqjFYDuPs6YCiJLqd9sgtPRESamgYTgrsPS/lc7e7/5e5h\nEomIiBSQdE9MG+fuo83sMeoYiObu34k0skZwd8zUrFGXhqoERUSS0jUq3xO8//+oDm5mdwEnA5+4\ne0bVUJWllVSur6SsvExJoRZ3p3J9JZWl6n0kIg1L98S0ecH7cxEefzJwG/DXTAtYWr4U1kPZurKc\nBdWcVJZWJv5GIiINSFdl9CZp5izKRS8jd59tZj2yKaOqRRWLOi3KNhQRkaKXrsro5OD9ouA9WYV0\nFnmc3M7MRgGjADp07ZCvw4qIFJ10VUYfApjZ8e5ekbLpF2Y2n8QcR5Fz90nAJEiMQ8jHMUVEilGY\n7qNmZkemLBwR8nsiIlJAwkxdcS7wFzPbI1j+LFgnIiLNSNqEYGYlwNfd/aBkQnD3Dbk6uJndDxwH\ndDSzlcAv3f3O9N8SEZEopE0I7l5tZlcCD+YyEaSUPyLXZYqISGbCtAXMMLOfm9l+ZlaefEUemYiI\n5FWYNoQzgveLUtY58NXchyMiYblD6uD82ssijdVgQnD3/fMRiIiEN3dyH/buuZb9B6zBLJEMqrcZ\n1sIpUR9AyVCYOwTMrA/QG6h5bqO7ZzzdhIhkzh327rmW7t/6hOrtUNIisb6kpePVulOQzDWYEMzs\nlyR6AvUGngBOAl4gi/mHRCRzZrD/gDVUb4fqbS0oKdle8wxDK1EykMyFubk8DRgMfOzu5wAHAXuk\n/4qIRMlsx51B6gNtlQwkG2ESwhZ3rwa2mVl74BNgv2jDEpF03KF6e3Jh5/UimQqTEF4zsz2B24F5\nwHzgxUijEpF6ucOylzoDRknL7TvdISTbEGonBiUKCSNML6MLg48TzexfQHt3fyPasESkPmbw8dJO\nAHX2Mtr2ZQkr5nfaadtnK8tos+dWWrerijl6acrSPQ/h4HTb3H1+NCGJSEOOGLlop95EZoleRgD/\n1eVnO+373bGzmf9wTw4evpTjLlqgdgapV7o7hJuD99ZAf+B1EjenfYHXgMOjDU1E0km9sI8uv7Te\n/R657hgAVizszMCLF0QdlhSwetsQ3H2guw8EVgMHu3t/dz8EqAA+yleAIpJeumSQzb5SfMIMTOvp\n7m8mF9x9kZl9M8KYRIpWuukoam/L9OKugWtSnzAJ4Q0zuwO4N1j+EaBGZZEcW/xUd/Y/bDWt239Z\n0xj8xcbdWPZyFz5b1Y6tlaU11T+ZumXdeGZNqKBVWRVHjNSzyGVnYRLCOcD/AZI/R2YDf4osIpEi\n5A5r3iln8TPdGXzxfPbsWslnK8v4+2WDWTKzR9blHzj0PXqf8AHP/bGipoFZdwpSW5hup18Avw9e\nIpJjtat+Xri9op49G2fc+vG4w6wJFcyb0pOX7ukDoN5GUq8wcxkdCYwBuqfu7+6a/lokC/lo4DWD\n4y5awPyHe9asUzKQ+oQZqXwncAtwFPCtlJeINFHj1o8HqLlDSDVrQoVGLkudwrQhbHD3JyOPRKSZ\niuNBNtXViWPMmlCx06C05DLoTkF2FSYhPGtmNwEPA1uTKzVSWaRhcyf3YWtlac3FN/mLvVVZtFNI\nTL/xUPp9710O/8ki+p+xmLKOW2uqj8q7bWDz+t2VDGQXYRLCYcF7/5R1DgzKfTgizYc7bK0s3ekX\nefIX+oqFnSM7bq9BH3DoiCWUtNhOq7IqWpXtPMVF31PeVzKQOoXpZTQwH4GINDfJX+QA8x/uWZMY\nDh6+NNKEMOiSeZS02M6e+26u88KvZCD1CfX0VTMbZmZXmtl1yVfUgYk0B6lJIan2cq7t1X1jvclA\nJJ0GE4KZTQTOAH5GYnK700l0QRUpamGeOVBfL58offphez77qK16EkmjhblDOMLdzwb+4+7Xk5jl\n9BvRhiXStM2d3Gen7pvJC//cyX1q9kmuS/byufyZBzh4+NKsp59oyMw/HEL19hZKCtJooR6hGbx/\nbmb7AFVAl+hCEmnaUhuLk0kheeHfWllacxE2g1ZlVTuNDI46GQAsmdmDV+7vxReVu6naSBolTC+j\nacEjNG8i8fhMB+6INCqRJixdY3Htvv2pD7LJ59TTQ698hZJQLYQiO4RJCDe6+1ZgiplNI/HAnC+i\nDUukaWvMlBCX7ZW/RJAcoSySiTAJ4UXgYIAgMWw1s/nJdSLN0ejyS9NeXOtrLM5HlVA6msFUspHu\nmcp7A/sCbcysgkQPI4D2wO55iE0kr2pX6dRerj0/UO0BZlGOLQhjv35rmDWhQlNSSMbS3SEMBUYC\nXUk8Xzn5T2wTcE20YYnkT9i6/ab++MmDhy+lVVmVkoFkrN6E4O53A3eb2ffdfUoeYxLJm6Z+kQ8r\n+ewDJQPJRph+CF3NrL0l3GFm883shMgjE5FQklVZSgaSrTAJ4Vx33wicAOwF/Bj4XS4ObmYnmtlS\nM/u3mV2VizJFRCQzYXoZJX93fBv4q7u/ZZb9bxEzawFMAI4HVgKvmtmj7r4427JFisHvPx2vuwLJ\nqTB3CPPM7CkSCWG6mbUDqnNw7EOBf7v7++7+JfAA8N0clCsSWiH32689VYZItsIkhPOAq4Bvufvn\nwG7AOTk49r7AipTllcE6kbwat358QSWG3386nu+Onb3LVBki2Uo7DsHdP3b3ahJTVgDg7p8Cn6bu\nE2WAZjYKGAXQoWuHKA8lRS7ZUyefI4sbkqwWcodnJ1Sw4OGe3DL4TKDuqTJEspGuDeEJGh6NHGaf\n+nwE7Jey3DVYtxN3nwRMAuhW0U2/hSRSZjuqkeLoklpfu4AZDLxoAQtCTJUhkql0VUYHmdnGNK9N\nQDZDM18FDjCz/c1sN+BM4NEsypMiNLr80mYzliA50rgxz1VQdZHkUrqBaS2iPLC7bzOzi4HpQAvg\nLnd/K8pjSvORbpqJQmoPAPju2Nk7PW8Zdv71X/u5Cun2FclGmG6nkXH3J0hUO4mEEuZuILlPISSG\nZDJInVK79vQTdT1Xob59RbJhXkD3nN0quvkVM6+IOwyJSSZVQ5kmhairoVInyku9oKebfqIx+4qk\nGl0+ep67929oPz1CQwpCPtsJ8pUMYNcLeroLfGP2FclEqCqjYFRx59T93X15VEGJ5EpTa3AuhGos\nKV4N3iGY2c+ANcDTwOPBa1rEcYnsRBdSkeiFuUO4FOgZDEgTkSwUUoO3FJ8wbQgrgA1RByLSkEKb\nYkKk0KSbuuLy4OP7wCwzexzYmtzu7rdEHJtInWr30Glq7QQihSpdlVG74H158NoteAEUTl9VabaS\nvWxS7xoKJTmMLr9UdzvS5KQbqXw9gJmd7u7/SN1mZqdHHZhIc6ZkIE1RmDaEq0OuE4mdLrQimUvX\nhnASiYfi7Gtmf0jZ1B7YFnVgIpnK52yl49aPZ+7kPmytLOWR644Jtb9IU5WuDWEVMA/4TvCetAm4\nLMqgRHIh120Lyecl1B4hfMTIRbjDwIsX1KyrrobLO16a9nsiTU26NoTXgdfN7D53r8pjTCI5l6tf\n5vVd1GuvLylJP0WFSFOUrsroTYLeRFbHv2Z37xtdWCIikm/pqoxODt4vCt7vCd7PQt1ORUSanXRV\nRh8CmNnx7p76qKZfmNl84KqogxMRkfwJ0+3UzOzIlIUjQn5PREQKSJjJ7c4D7jKzPQAD/gOcG2lU\nIiKSdw0mBHefBxwUJATcXRPdiYg0Q+l6GZ3l7vemTHKXXA9ocjsRkeYm3R1C2+C9XZp9RESkmUjX\ny+jPwccb3P2LPMUjIiIxCdOovMjM1gDPB68X1I4gItL8NNh91N2/DowA3gSGkZjOYmHUgYmISH41\neIdgZl2BI4GjgYOAt4AXIo5LRETyLEyV0XLgVeC37n5BxPGIiEhMwiSECuAo4IdmdhXwLvCcu98Z\naWQSmTMH3UybdZW7rN/SsYwHZl4RQ0Qi0hSEGZj2upm9B7xHotroLOBYQAmhQLVZV8mWjmV1rheR\n4hWmDeE1oBUwl0Qvo2OSE9+JiEjzEabK6CR3Xxt5JCIiEqsw3U6VDEREioCmsRYRESBclZE0M1s6\nltU0ILdds5EWVdtrtl1aPpqNXTuox5FIEUo32+nwdF9094dzH47kQ+qF/tLy0btsb7/yP6xfqTkN\nRYpNujuEU9JscyDjhGBmpwNjgG8Ch7r7a5mWJZmpKxGk6sFyLi0fzfj14/IUkYjELd1sp+dEeNxF\nwHDgzw3tKLnXUDIQkeIUqg3BzIYBBwKtk+vcfWymB3X3t4NyMy1CMpBJIhhdfinj1o+PIBoRaWrC\nDEybCOwODATuAE4DXok4LmmkdNNRdHp9JSXuGZXrGBv7dtilTDU4izQ/Ye4QjnD3vmb2hrtfb2Y3\nA0829CUzmwHsXcema939kbABmtkoYBRAh64dGti7eNU3HUXnhSuyLrt2uZriQqR5CpMQtgTvn5vZ\nPsCnQJeGvuTuQ7IJLKWcScAkgG4V3TL7mVukcpEMRKR4hBmYNs3M9gRuAuYDHwD3RxmUZG/Fwq/k\npJw1/fbLSTki0vSFuUO40d23AlPMbBqJhuWsnrFsZqcCtwKdgMfNbKG7D82mTNmh88IVdEZ3ByLS\nOGHuEF5MfnD3rcHzlF9Ms3+D3H2qu3d191bu3lnJIDdWLPxKTquJNqrNRqSopBupvDewL9DGzCqA\nZB/R9iR6HUkT0n7lf+jPvJyUVUULaFOy0xQXqepqvBaRwpeuymgoMBLoCtySsn4jcE2EMUkj5XKg\nWbLNoM26SnUtFSky6UYq3w3cbWbfd/cpeYxJQtKIYxHJpTBtCHPM7E4zexLAzHqb2XkRxyUiInkW\nJiH8BZgO7BMsvwPop2nMoro7yFV3VREpPGG6nXZ09wfN7GoAd99mZtsb+pJkL910FFHpzzxYCNtL\nW6Q9vtoXRJqfMAlhs5ntRWLKa8xsALAh0qgEiHY6ioZsq2pR7/E1dYVI8xQmIVwOPAp8zczmkBhM\ndlqkUUnsWvElq1fuTZuOmi1EpFg0mBDcfb6ZHQv0JDEWYam7V0UemexixcKv5GysQRg9WM4aNHWF\nSLEIM/11a+BC4CgS1UbPm9lEd89q+gppvHwmA9BIZZFiE6aX0V9JPBznVuC24PM9UQYlu8r3zKUf\n0C2vxxOR+IVpQ+jj7r1Tlp81s8VRBSQ71Dd1RNQ+oBtt+jlb0NQVIsUkTEKYb2YD3P0lADM7DHgt\n2rAEqOnamc8RyePXj8vbsUSkaQmTEA4B5prZ8mC5G7DUzN4E3N37Rhad5JWSgUhxC5MQTow8Cklr\n/Ppxkd4lKBGICITrdvphPgKR9AzHa2Ygzx0lAxFJCnOHIE3AuPXj2di3A++s/Oou2zqylv3ZkbfD\nJA4NNxOR2pQQCsiWjmV8g/frXD9u5via5TWD9tupd1C7lf/JS3wiUtiUEApI2AnlNPGciGQizMA0\nEREpAkoIIiICKCGIiEhACUFERAAlBBERCSghiIgIoIQgIiIBJQQREQGUEEREJKCEICIigBKCiIgE\nlBBERARQQhARkYASgoiIAEoIIiISiCUhmNlNZrbEzN4ws6lmtmcccYiIyA5x3SE8DfRx977AO8DV\nMcUhIiKBWBKCuz/l7tuCxZeArnHEISIiOzSFNoRzgSfjDkJEpNhF9kxlM5sB7F3Hpmvd/ZFgn2uB\nbcB9acoZBYwC6NC1QwSRiogIRJgQ3H1Iuu1mNhI4GRjs7p6mnEnAJIBuFd3q3U9ERLITWUJIx8xO\nBK4EjnX3z+OIQUREdhZXG8JtQDvgaTNbaGYTY4pDREQCsdwhuPvX4ziuiIjUryn0MhIRkSZACUFE\nRAAlBBERCSghiIgIoIQgIiIBJQQREQGUEEREJKCEICIigBKCiIgElBBERARQQhARkYASgoiIAEoI\nIiISUEIQEREALM3DypocM1sLfBh3HCk6AuviDiJGxXz+xXzuUNznX4jn3t3dOzW0U0ElhKbGzF5z\n9/5xxxGXYj7/Yj53KO7zb87nriojEREBlBBERCSghJCdSXEHELNiPv9iPnco7vNvtueuNgQREQF0\nhyAiIgElhCyZ2U1mtsTM3jCzqWa2Z9wx5ZOZnW5mb5lZtZk1y54XtZnZiWa21Mz+bWZXxR1PPpnZ\nXWb2iZktijuWfDOz/czsWTNbHPybvzTumHJNCSF7TwN93L0v8A5wdczx5NsiYDgwO+5A8sHMWgAT\ngJOA3sAIM+sdb1R5NRk4Me4gYrINuMLdewMDgIua2397JYQsuftT7r4tWHwJ6BpnPPnm7m+7+9K4\n48ijQ4F/u/v77v4l8ADw3Zhjyht3nw2sjzuOOLj7anefH3zeBLwN7BtvVLmlhJBb5wJPxh2ERGpf\nYEXK8kqa2UVBGmZmPYAK4OV4I8mtlnEHUAjMbAawdx2brnX3R4J9riVxS3lfPmPLhzDnL1IszKwM\nmAKMdveNcceTS0oIIbj7kHTbzWwkcDIw2JthP96Gzr/IfATsl7LcNVgnRcDMSkkkg/vc/eG448k1\nVRllycxOBK4EvuPun8cdj0TuVeAAM9vfzHYDzgQejTkmyQMzM+BO4G13vyXueKKghJC924B2wNNm\nttDMJsYdUD6Z2almthI4HHjczKbHHVOUgg4EFwPTSTQqPujub8UbVf6Y2f3Ai0BPM1tpZufFHVMe\nHQn8GBgU/L++0My+HXdQuaSRyiIiAugOQUREAkoIIiICKCGIiEhACUFERAAlBBERCSghSN6Y2Ugz\n2yfEfpPN7LSw63MQ1zUpn3uEmckziGWZmV2QZp9+ueyWGPz9bsuyjFnJWWnN7IlsZ+c1s+PMbFrw\n+YxgBthp2ZQp8VFCkHwaCTSYEGJwTcO71Om/3D3duJN+QGz91M0s7UwE7v5td/8sV8dz978D5+eq\nPMk/JQTJSPBLeomZ3Wdmb5vZQ2a2e7DtEDN7zszmmdl0M+sS/LLvD9wXDOhpY2bXmdmrZrbIzCYF\nI0HDHn+XYwTrZ5nZDWb2ipm9Y2ZHB+t3N7MHg7nsp5rZy2bW38x+B7QJYkrOQ9XCzG4P5rx/ysza\nhIjn9OA8Xjez2cEo5rHAGUHZZ5jZoWb2opktMLO5ZtYz+O5IM3vYzP5lZu+a2Y0p5Z4TnMcrJAZG\nJdefEpzDAjObYWadg/VjzOweM5sD3BP8nR8I/htNBdqklPGBmXU0swtSBlotM7Nng+0nBPHON7N/\nWGIOn+TzIJaY2XwSU59Lc+HueunV6BfQA3DgyGD5LuDnQCkwF+gUrD8DuCv4PAvon1JGecrne4BT\ngs+TgdPqOOZk4LQQx7g5+PxtYEbw+efAn4PPfUhMRNg/WK6sdV7bgH7B8oPAWfXFkrL8JrBv8HnP\n4H0kcFvKPu2BlsHnIcCUlP3eB/YAWgMfkpgvqQuwHOgE7AbMSZYHdGDHwNLzU855DDAPaBMsX57y\nt+lb67w/ADqmxFcKPA+cAnQk8YyLtsG2XwDXBfGtAA4ALPj7TEsp47jUZb0K66XJ7SQbK9x9TvD5\nXuAS4F8kLrhPBz/4WwCr6/n+QDO7EtgdKAfeAh4LcdyeDRwjOenYPBIXeICjgPEA7r7IzN5IU/4y\nd19YRxmPS6muAAACuElEQVTpzAEmm9mDKcevbQ/gbjM7gEQyLU3Z9oy7bwAws8VAdxIX5VnuvjZY\n/3fgG8H+XYG/B3dGuwHLUsp61N23BJ+PAf4A4O5vNHDe44GZ7v6YmZ1M4gFAc4K/8W4kpqzoReLv\n824Q073AqDRlSgFRQpBs1J73xEn8anzL3Q9P90Uzaw38kcSv1RVmNobEr88wGjrG1uB9O5n9G9+a\n8nk7KdUs9XH3C8zsMGAYMM/MDqljt18Bz7r7qZaYT39WmmM2FPetwC3u/qiZHUfiziBpc0Px1maJ\nGXu7k5inCRJ/46fdfUSt/fo1tmwpHGpDkGx0M7PkRfmHwAvAUqBTcr2ZlZrZgcE+m0hMBAg7Lv7r\ngrrpxvQeSneM+swBfhDs3xv4XynbqiwxrXHGzOxr7v6yu18HrCVR5ZN6vpC4Q0hOlT0yRLEvA8ea\n2V5BfKfXU9ZP0pQxm8R/G8ysD4lqo9qxH0KiSu0sd68OVr8EHGlmXw/2aWtm3wCWAD3M7GvBfiNq\nlyeFSwlBsrGUxHNl3yZRp/0nTzxW8jTgBjN7HVgIHBHsPxmYaGYLSfwivp3EM5mnk5hWOpQGjlGf\nP5JIIouBX5OontoQbJsEvJHSqJyJm8zsTUt0WZ0LvA48C/RONioDNwL/z8wWEOLOxd1Xk/jl/yKJ\nhPZ2yuYxwD/MbB6wLk0xfwLKgv9GY0lUgdV2MYkqu2eDWO8IqqlGAvcH1UwvAr3c/QsSVUSPB43K\nnzR0HlI4NNupZCSo8pjm7n1iDiUUM2sBlLr7F8Gv2xlAzyC5ZFLeZBLn/1AOwyx4QfXVz9395Lhj\nkcZTG4IUi91J/AIuJVE/fmGmySCwAfiVmXX09GMRikZwF/RL6r4LkQKgOwQREQHUhiAiIgElBBER\nAZQQREQkoIQgIiKAEoKIiASUEEREBID/AfqZhciLKC59AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x11530ec90>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_decision_regions(X_combined_std, y_combined, classifier=svm, test_idx=range(105, 150))\n",
    "plt.xlabel('petal length [standardized]')\n",
    "plt.ylabel('petal width [standardized]')\n",
    "plt.legend(loc='upper left')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "\n",
    "虽然gamma值较大的模型对训练集分类效果很大，但其泛化能力一般很差，所以选择适当的gamma值有助于避免过拟合。\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## 决策树学习\n",
    "\n",
    "\n",
    "如果我们在意模型的可解释性，那么决策树(decision tree)分类器绝对是上佳的选择。如同名字的字面意思，我们可以把决策树理解为基于一系列问题对数据做出的分割选择。\n",
    "\n",
    "举一个简单的例子，我们使用决策树来决定某一天的活动：\n",
    "\n",
    "\n",
    "\n",
    "![](https://ooo.0o0.ooo/2016/06/21/5769da1318edf.png)\n",
    "\n",
    "基于训练集中的特征，决策树模型提出了一系列问题来推测样本的类别。虽然上图中做出的每个决策都是根据离散变量，但也可以用于连续型变量，比如，对于Iris中sepal width这一取值为实数的特征，我们可以问“sepal width是否大于2.8cm？”\n",
    "\n",
    "\n",
    "训练决策树模型时，我们从根节点出发，使用信息增益(information gain, IG)最大的特征对数据分割。然后迭代此过程。显然，决策树的生成是一个递归过程，在决策树基本算法中，有三种情形会导致递归返回:（1）当前节点包含的样本全属于同一类别，无需划分；（2）当前属性集为空，或是所有样本在所有属性上取值相同，无法划分；（3）当前节点包含的样本集合为空，不能划分。\n",
    "\n",
    "每一个节点的样本都属于同一个类，同时这也可能导致树的深度很大，节点很多，很容易引起过拟合。因此，剪枝操作是必不可少的，来控制树深度。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## 最大信息增益\n",
    "\n",
    "为了使用最大信息增益的特征分割数据，我们需要定义一个在决策树学习过程中的目标函数。此处，我们的目标函数是在每一次分割时最大化信息增益，我们定义如下：\n",
    "\n",
    "![](https://ooo.0o0.ooo/2016/06/21/5769dd8f3d6bb.png)\n",
    "\n",
    "其中，f是具体的特征，$D_{p}和D_{j}$是当前数据集和用特征f分割后第j个子节点的数据集，I是某种度量，$N_{p}$是当前数据集样本个数，$N_{j}$是第j个子节点的数据集中样本个数。为了简化和减小搜索空间，大多数决策树(包括sklearn)都是用二叉树实现的。这意味着每一个父节点被分割为两个子节点，$D_{left}和D_{right}$:\n",
    "\n",
    "![](https://ooo.0o0.ooo/2016/06/21/5769fb190d9c6.png)\n",
    "\n",
    "\n",
    "常用的度量I包括基尼指数(Gini index, $I_{G}$)、熵(Entropy, $I_{H}$)和分类错误(classification error, $I_{E}$)。我们以熵为例：\n",
    "\n",
    "![](https://ooo.0o0.ooo/2016/06/21/5769fbcbe1c78.png)\n",
    "\n",
    "此处,$p(i|t)$指的是在节点t中属于类别c的样本占比。所以如果某个节点中所有样本都属于同一个类，则熵为0，如果样本的类别时均匀分布，则熵最大。比如，在二分类情况下，如果p(i=1|t)=1或p(i=0|t)=0，则熵为0.因此，我们说熵的评价标准目的是最大化树中的互信息。\n",
    "\n",
    "\n",
    "基尼系数可以被理解为最小化误分类的概率：\n",
    "\n",
    "![](https://ooo.0o0.ooo/2016/06/21/5769fcdb6fb31.png)\n",
    "\n",
    "和熵一样，如果节点中样本的类别均匀，则基尼系数最大，比如，在二分类情况下：\n",
    "\n",
    "![](https://ooo.0o0.ooo/2016/06/21/5769fd2f84f95.png)\n",
    "\n",
    "\n",
    "通常，熵和基尼系数的结果相似，所以不需要花太多时间在选择度量上面。\n",
    "\n",
    "\n",
    "另一种常用的度量是分类误差：\n",
    "\n",
    "![](https://ooo.0o0.ooo/2016/06/21/5769fd97f352e.png)\n",
    "\n",
    "这个度量更建议在剪枝时使用，而不是在构建决策树时使用。举个简单的例子，说明一下为什么不建议构建树时用：\n",
    "\n",
    "![](https://ooo.0o0.ooo/2016/06/21/5769fe06df61a.png)\n",
    "\n",
    "$D_{p}$中包含40个正例样本和40个负例样本，然后分割为两个子节点。信息增益使用分类误差作为度量，得到的值在A、B情况下相同，都是0.25,计算过程如下：\n",
    "\n",
    "![](https://ooo.0o0.ooo/2016/06/21/5769fea2248cb.png)\n",
    "\n",
    "如果使用基尼系数，则会按照B情况分割：\n",
    "\n",
    "\n",
    "![](https://ooo.0o0.ooo/2016/06/21/5769fedf9a5ff.png)\n",
    "\n",
    "同样，如果用熵作为度量，也会按照B分割：\n",
    "\n",
    "\n",
    "![](https://ooo.0o0.ooo/2016/06/21/5769ff05d59bb.png)\n",
    "\n",
    "![](https://ooo.0o0.ooo/2016/06/21/5769ff167e10c.png)\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 构建一棵决策树\n",
    "\n",
    "决策树通过将特征空间分割为矩形，所以其决策界很复杂。但是要知道过大的树深度会导致过拟合，所以决策界并不是越复杂越好。我们调用sklearn，使用熵作为度量，训练一颗最大深度为3的决策树。还有一点，**对于决策树算法来说，特征缩放并不是必须的**。代码如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.tree import DecisionTreeClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "tree = DecisionTreeClassifier(criterion='entropy', max_depth=3, random_state=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DecisionTreeClassifier(class_weight=None, criterion='entropy', max_depth=3,\n",
       "            max_features=None, max_leaf_nodes=None,\n",
       "            min_impurity_split=1e-07, min_samples_leaf=1,\n",
       "            min_samples_split=2, min_weight_fraction_leaf=0.0,\n",
       "            presort=False, random_state=0, splitter='best')"
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tree.fit(X_train, y_train) # 这里不需要标准化数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X_combined = np.vstack((X_train, X_test))\n",
    "y_combined = np.hstack((y_train, y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY0AAAEKCAYAAADuEgmxAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XmYVNW57/HvD2gFaRERDiIomITAUa8B5CDOouAQjEZj\noibGIRqOVz0GNSeDuceoGW40N0ajJgSHOMRoEocEjYY4EecBEAURohFREEci2AgI9Hv/2LuwaLqr\nN11VXdXdv8/z1FN7WLXXW43223uttddSRGBmZpZFp0oHYGZmbYeThpmZZeakYWZmmTlpmJlZZk4a\nZmaWmZOGmZll5qRhZmaZOWmYmVlmThpmZpZZl0oHUGq129RGrx16VToMM7M25fVZr78bEX2aK9fu\nkkavHXpx7oPnVjoMM7M2ZWKviQuzlHPzlJmZZeakYWZmmTlpmJlZZu2uT6MxNetqGLJ0CLVraisd\nStWqq6ljfq/5rOm8ptKhmFkV6xBJY8jSIQzsOZDarWuRVOlwqk5EULe0DpbCnD5zKh2OmVWxDtE8\nVbum1gmjAEnU9qr1nZiZNatDJA3ACaMZ/vmYWRYdJmmYmVnxnDRa0YNTH2Svnfdi9NDRXHHJFZUO\nx8xskzlptJJ169bx3bO+y+/u+h0PP/8wd956J/Pnzq90WGZmm6RDjJ7aFL1GH0ynd97b6Hh9n21Y\n+uTUFl/32aefZcdP7sjATwwE4PPHfJ6pd01lyE5DWnxNM7PW5qTRQKd33qO+98YTHjaWSDbFkjeW\nsN2A7dbv9+vfj5lPzyzqmmZmrc3NU2ZmlpmTRivpt10/3lj0xvr9JYuX0K9/vwpGZGa26Zw0Wsmw\n/xjGKy+/wsIFC/noo4/40+//xEGHHVTpsMzMNon7NFpJly5d+PHlP+a48cexbt06jjvpOIbuPLTS\nYZmZbRInjQbq+2zT5OipYo09dCxjDx1b9HXMzCrFSaOBYobVmpm1d+7TMDOzzCqWNCR1lfS0pOck\nvSDpwkbKbC7p95JelvSUpEGtH6mZmeVU8k5jNXBARHwGGAYcIml0gzKnAP+KiE8BPwcubuUYzcws\nT8WSRiTq0t2a9BUNih0B3JBu3wYcKM/hbWZWMRXt05DUWdIs4G3gvoh4qkGR/sDrABGxFlgGbDSM\nSdIESdMlTa97t67haTMzK5GKJo2IWBcRw4ABwChJu7TwOpMjYmREjKztXb2rz008dSI7b7cz+w3b\nr9KhmJm1SFWMnoqI94GHgEManFoMbA8gqQuwFVDczIGZ4im831LHnHgMt9x9S2kuZmZWAZUcPdVH\nUs90uxswDpjXoNgU4MR0+2jgwYhS/Qpv3E1Xd2PSZVusTxQRMOmyLbjp6m5FX3uPffagZ6+eRV/H\nzKxSKnmn0Q94SNLzwDMkfRp3S7pI0uFpmWuBbSS9DJwDfKecAUVAXZ2489au6xPHpMu24M5bu1JX\np5LdcZiZtVUVeyI8Ip4Hhjdy/Py87VXAF1srJglOm/ghAHfe2pU7b+0KwJHHruK0iR/icVtm1tFV\nRZ9GNclPHDlOGGZmCSeNBnJNUvny+zjMzDoyJ408+X0YRx67iqlPLuXIY1dt0MdRjNOOP43D9jmM\nf87/J8MHDed31/2uNIGbmbUSz3KbR4La2tigDyPXVFVbG0U3UU367aQSRGlmVjlOGg189esriWB9\ngsglDvdpmJm5eapRDROEE4aZWcJJw8zMMnPSMDOzzJw0zMwsMycNMzPLzEmjlSx+fTFHjT2KfXbd\nh30/sy9X/+LqSodkZrbJPOS2lXTp0oULLrmAXUfsSt0HdRy0+0HsO3Zfhuw0pNKhmZll5qTRiDfW\nvMG8NfNYUb+C7p26M7RmKNvVbFfUNfv260vffn0BqN2ylsFDB/PmG286aZhZm+Kk0cAba95gxuoZ\ndKITNdSwsn4lM1bPACg6ceS89uprzJk1hxGjRpTkemZmraXJpCHpqAyfXxUR95Qwnoqbt2YenehE\nFyU/mi50YW2sZd6aeSVJGivqVnDql07lop9dxJY9tiz6emZmranQncbVwJ+BQs9D7wu0KGlI2h64\nEegLBDA5Ii5vUGb/NIYF6aE7IuKiltSX1Yr6FdRQs8GxznRmRf2Koq+9Zs0aTvnSKRx13FGMP3J8\n0dczM2tthZLGvRHxtUIflvTbIupeC5wbETMlbQnMkHRfRMxtUO6RiDisiHo2SfdO3VlZv5IueT+a\ndayje6fuRV03Ijj762czeOhgTjv7tGLDNDOriCaH3EbE8c19OEuZAp9dEhEz0+0PgBeB/i29XqkM\nrRlKPfWsjbVEBGtjLfXUM7RmaFHXffqxp7nt5tt49KFHOXC3AzlwtwO5/977SxS1mVnraHGfRkTc\nUaogJA0iWfr1qUZO7yHpOeAN4JsR8UKp6m1Mrt+i1KOndt97d95c82YpQjQzq5hCzVOfS9//DdgT\neDDdHwM8DpQkaUiqBW4HJkbE8ganZwIDI6JO0meBPwGDG7nGBGACwNYDti46pu1qtivZSCkzs/ak\nUPPUyRFxMlAD7BQRX4iILwA7p8eKJqmGJGHc3NidS0Qsj4i6dPseoEZS70bKTY6IkRExsrZ3bSlC\nMzOzRmSZRmT7iFiSt/8WsEOxFUsScC3wYkRc2kSZbdNySBqVxvtesXWbmVnLZHm47wFJU4Fb0v1j\ngFL04O4FfBWYLWlWeuw80oQUEZOAo4H/LWktsBI4NqLYlbrNzKylmk0aEXGmpCNJnsmA5HmKO4ut\nOCIepfAzIETElcCVxdZlZmalkXUakZnABxFxv6QtJG2ZDpM1M7MOpNk+DUlfB24Dfp0e6k8yisky\nWvb+Mn7zq9+0+POTL5/Mhx9+WHQcj/39MZ55/Jmir2NmHVeWjvAzSPoflgNExEskw3Ato2XvL+P6\nX1/f4s9PvmIyKz9cWXQcj//9cZ550knDzFouS9JYHREf5XYkdSGZK8oy+tH3fsTCfy7kwN0O5MJv\nXwjAVT+7ioNHH8yY4WO45MJLAFixYgVfOfwrHDDiAPYbth9/+sOfuOaKa3jrjbf4wtgvcNTYjZ+3\n/OF5P2SfXfdhzPAxXPCtCwB49513OeVLp3Dw6IM5ePTBPP3Y07z26mvcOPlGJl8+mQN3O5AnH32y\n1b6/mbUfWfo0/i7pPKCbpHHA6cBd5Q2rffnej77HvBfm8cCMBwCYdt80Fry0gL8+8VcighOOPIEn\nHnmC9955j237bcvNU24GYPmy5fTYqgeTLp/E7fffzja9t9ngukvfW8q9f76XR+c8iiSWvb8MgP85\n53+YcNYEdt97dxa9tojjxh/HI7Mf4YQJJ9C9tjunn3N66/4AzKzdyJI0vgOcAswG/hO4JyK8VmkR\npt03jWn3T2PsyLFAcoex4KUF7L737lzwrQv4wXd/wLjx4xi99+iC1+mxVQ8233xzzv762YwbP45x\n48cB8PADD/OPuf9YX+6D5R+woq74WXrNzLIkja8At+YnCkmHRcTd5QurfYsIzvrWWZww4YSNzt33\n9H08cO8D/OT8n7DPAftw7v85t8nrdOnShXufuJdHHnyEu2+/m+t+eR2333c7UR/85bG/0LVr13J+\nDTPrgLL0aVwBPCLp3/OOlXVNi/amdsta6j6oW78/5qAx3HL9Lev/+l+yeAnvvP0Ob77xJt226MbR\nXzma0889ndnPzk4+X7vh53NW1K1g+bLljD10LBf97CLmPp/MKr/fuP249qpr15ebM2tOo3GYmW2q\nLHcaC0iap26TdEFE/JFmHsqzDfXaphej9hzFfsP244CDD+D7F3+fl158ifF7Jwsxda/tzlU3XMWC\nfy7gom9fRKdOnehS04WLr7wYgONPPZ4vH/Zl+vbryx33fzxFV90HdZx41ImsXrWaiOCCn14AwA9/\n/kO+e9Z3GTN8DGvXrWWPvffgkl9ewkHjD+LUY05l6pSp/OjyHzXb/GVm1pCam5VD0syIGJFOFHgL\n8BxwUETs2hoBbqodhu8Q5z64YZPO6DdGM2jIoMoE1Ia8Ov9VntzOo6rMOqKJvSbOiIiRzZXL0jy1\nBCAi3gUOJhluu0tx4ZmZWVvUbNKIiPF52/UR8d8RkSXZmJlZO1No5b7LImKipLto5GG+iDi8rJGV\nWESQzrJujfDkwWaWRaGO8JvS9//XGoGUU11NHXVL66jtVevE0YiIoG5pHXU1HlllZoU1mTQiYkb6\n/vfWC6c85veaD0uh9l2v6teUupq65OdkZlZAoeap2RSYY6paR081Zk3nNczpM6fSYZiZtXmFmqcO\nS9/PSN9zzVXHU4IJCyVtD9wI9E2vNzkiLm9QRsDlwGeBD4GTImJmsXWbmVnLFGqeWgggaVxEDM87\n9W1JM0nmpCrGWuDciJgpaUtghqT7ImJuXplDgcHpa3fgV+m7mZlVQJahs5K0V97Onhk/V1BELMnd\nNaSrAL5IssBTviOAGyPxJNBTUr9i6zYzs5bJMo3I14DfSNoq3X8/PVYykgYBw4GnGpzqD7yet78o\nPbaklPWbmVk2BZOGpE7ApyLiM7mkERHLShmApFrgdmBiRCxv4TUmABMAth6wdQmjMzOzfAWbmSKi\nHvhWur2sDAmjhiRh3BwRdzRSZDGwfd7+gPRYwzgnR8TIiBhZ29vDas3MyiVL38T9kr4paXtJvXKv\nYitOR0ZdC7wYEZc2UWwKcIISo4FlEeGmKTOzCsnSp3FM+n5G3rEAPlFk3XsBXwVmS5qVHjsP2AEg\nIiYB95AMt32ZZMjtyUXWaWZmRWg2aUTEjuWoOCIepZl1OSKZEOmMQmXMzKz1ZLnTQNIuwE7A+vVD\nI+LGcgVlZmbVqdmkIen7wP4kSeMekgfuHiV5mtvM2qAIyJ+7s+G+WVOydIQfDRwIvBkRJwOfAbYq\n/BEzq1aPX78LC57sS242/AhY8GRfFs7oU9nArE3I0jy1MiLqJa2V1AN4mw2HwZpZGxEB2w55h4H/\n8Tb166BTZ6hfB4NGvYXkOw5rXpakMV1ST+BqYAZQBzxR1qjMrCwk2HH0W9Svg/q1nalfmxzvsvm6\n9efNCskyeur0dHOSpL8CPSLi+fKGZWblIqV3GGs3Pm7WnELraYwodM5TlJu1TRFJk1Rjxwt1jrvp\nyqDwncbP0veuwEjgOZLnKnYFpgN7lDc0Myu1XKf3wP94m05d1q3v04j6JCEEyfu6tWmC6MT6vo5V\nyzcjArbo+VGlv4ZVUJOjpyJiTESMIZlRdkQ6t9NuJLPRbjT/k5lVPwnenN+Hhc/8G506f9xU9erT\nfXnt2T7rE0T92s48dv0uTLtqOBEw7arh/OXHo1n9wWbrR11Zx5SlI3xIRMzO7UTEHEn/XsaYzKyM\n9jxpzgZNTbnO8fz9LpuvY+3KGmbeMYSZdwwBYMRR8+k5oM5NVB1cluc0npd0jaT909fVgDvCzdqw\nhr/4G9vf/4xnNzi2/xnPOmFYpqRxMvAC8I30NRdPHGhWVRo2GW3qfmPXm3bV8A2OPXTl8A0+52aq\njqnZpBERqyLi5xFxZPr6eUSsao3gzKx5c/82cH0nNXzcaT33bwOB5AnwXN9E7vy0q4bz+PW7NHq9\n3PmZdwxhxFHzOeeBW+m1wzL+/uvh6xNHc9ew9ivL3FN7ARcAA/PLR0SxU6OXxeuz/o2Jvb5R6TDM\nWtXQA17lgLNmsM3A5by3sAcP/mI35j04aIMyfz5/340+94dzDmz0el+69AFGHDV/fRPVwJFv8ur0\nfiyc0Q94doOk4qG4HYuimXtMSfOAs0meBl8/ujsi3itvaC0jjYxkRLCZtdRlSy/fIBlEwENXDefZ\ntFMcWJ9UnDDah4m9Js6IiJHNlcvSp7EsIu6NiLcj4r3cqwQxmlkVy08GEoxxx7iRLWk8JOmnkvaQ\nNCL3KkXlkq6T9LakOU2c31/SMkmz0tf5pajXzJpXX//xdmMd4/n9JNZxZHlOY/f0Pf+2JYADSlD/\n9cCVFF6b45GIOKwEdZlZRvX1MPWSUQz7/EtsO+RfvL+oli7d1vC1G+6m54A63l9UywNXjmDaVcN9\nx9HBZJmwcEy5Ko+IhyUNKtf1zaxlpl4yilHHzaNT56Qbs9vWK9nrpDkonVak54A6xp/3JAue6ueE\n0cFkXe51PLAzGy73elG5gmpgD0nPAW8A34yIF1qpXrMOa/Dei+nUeR09+69Agq616zZ6irxrj4/Y\n6aCFlQ3UWl2WIbeTgC2AMcA1JCv5PV3muHJmAgMjok7SZ4E/AYMbiXECMCHZ26GVQjNrv7beYfn6\nhJHT3FPk1jFk6QjfMyJOAP4VEReSzG776fKGlYiI5RFRl27fA9RI6t1IucnphIojwUtWmhXrX6/1\n4P3F3d3RbRvJkjRWpu8fStoOWAP0K19IH5O0rZT8PSNpFEm8Hu5rVmYvPdqf+nWdnThsI1mSxt3p\ncq8/JWkuehW4pRSVS7qFZOnYIZIWSTpF0mmSTkuLHA3MSfs0fgEcG809jWhmRTv4W0/z9C1DWVW3\nmZuhbANZngjfPCJW57ZJOsNX5Y5VGz8Rbla8y5ZeTn09dMryZ6W1C6V8IvyJ3EZErI6IZfnHzKx9\ncsKwxhRaI3xboD/QTdJwkqVeAXqQjKYyM7MOptCQ24OBk4ABJOuF55LGB8B55Q3LzMyqUZNJIyJu\nAG6Q9IWIuL0VYzIzsyqVpdVygKQeSlwjaaakg8oemZmZVZ0sSeNrEbEcOAjYBvgq8JOyRmVmZlUp\nS9LI9WV8FrgxnfvJI7fNzDqgLEljhqS/kSSNqZK2BOqb+YyZmbVDWWa5PQUYBrwSER9K2gY4ubxh\nmZlZNWryTiN9ToOIqI+ImRHxfrr/XkQ8n1/GzMw6hkLNU/dk+HyWMmZm1k4Uap76jKTlBc4LKHTe\nzMzamUIP93VuzUDMzKz6eUoyMzPLzEnDzMwyc9IwM7PMMiUNSZ0lbSdph9yrFJVLuk7S25LmNHFe\nkn4h6WVJz0saUYp6zcysZZpNGpL+C3gLuA/4S/q6u0T1Xw8cUuD8ocDg9DUB+FWJ6jUzsxbI8kT4\nN4AhEfFeqSuPiIclDSpQ5AiS+a4CeFJST0n9ImJJqWMxM7PmZWmeeh1YVu5AmtA/rT9nUXpsA5Im\nSJouaTq802rBmZl1NIWWez0n3XwFmCbpL8Dq3PmIuLTMsWUWEZOByQDSyKhwOGZm7Vah5qkt0/fX\n0tdm6QugtX4xLwa2z9sfkB4zM7MKKPRE+IUAkr4YEX/MPyfpi+UOLDUFOFPSrcDuwDL3Z5iZVU6W\nPo3vZjy2ySTdAjwBDJG0SNIpkk6TdFpa5B6S5rGXgauB00tRr5mZtUyhPo1DSRZe6i/pF3mnegBr\nS1F5RBzXzPkAzihFXWZmVrxCfRpvADOAw9P3nA+As8sZlJmZVadCfRrPAc9Jujki1rRiTGZmVqUK\nNU/NJh0lJWmj8xGxa/nCMjOzalSoeeqw9D3Xp3BT+n48rTfk1szMqkih5qmFAJLGRcTwvFPfljQT\n+E65gzMzs+qSZcitJO2Vt7Nnxs+ZmVk7k2XCwlOA6yRtRbIu+L+Ar5U1KjMzq0pKHoXIUDBJGkRE\npSYvzGSkFNMrHYRZOyB3XXYwmhERI5srVWj01PER8du8iQtzx4HqmrDQzEov2HjUpLVfWf+1CzVP\ndU/ftyxQxszMOpBCo6d+nW5eHBGrWikeMzOrYlk6wudIegt4JH09Wu39GmZmVh7NDp2NiE8BxwGz\ngfEkU4vMKndgZmZWfZq905A0ANgL2Af4DPAC8GiZ4zIzsyqUpXnqNeAZ4McRcVpzhc3MrP3K8mT3\ncOBG4MuSnpB0o6RTyhyXmZlVoSx9Gs8BNwC/AR4E9gPOL0Xlkg6RNF/Sy5I2mstK0kmS3pE0K32d\nWop6zcysZbL0aUwHNgceJxk9tW9uMsNiSOoMXAWMAxYBz0iaEhFzGxT9fUScWWx9ZmZWvCx9GodG\nxDtlqHsU8HJEvAIg6VbgCKBh0jAzsyqRpXmqHAkDoD/wet7+ovRYQ1+Q9Lyk2yRt39iFJE2QNF3S\n9HIFa2Zm1T/F+V3AoHSVwPtI+lY2EhGTI2JkRIzs06rhmZl1LJVMGouB/DuHAemx9SLivYhYne5e\nA+zWSrGZmVkjCs1ye1ShD0bEHUXW/QwwWNKOJMniWODLDWLoFxFL0t3DgReLrNPMzIpQqCP8cwXO\nBVBU0oiItZLOBKYCnYHrIuIFSRcB0yNiCnCWpMOBtcBS4KRi6jQzs+JkXoSprfAiTGZmm05Q3CJM\nG1xMGg/sDHTNHYuIi1oenpmZtUXNdoRLmgQcA/wXyeJOXwQGljkuMzOrQllGT+0ZEScA/4qIC4E9\ngE+XNywzM6tGWZLGyvT9Q0nbAWuAfuULyczMqlWWPo27JfUEfgrMJBk5dU1ZozIzs6rU7OgpSZvn\nHrCTtDlJZ/iqvIfuqopHT5mZbbqso6eyNE89kduIiNXp+uBPFChvZmbtVKEnwrclmUCwm6ThJCOn\nAHoAW7RCbGZmVmUK9WkcTPIE9gDg0rzjy4HzyhiTmZlVqSaTRkTcANwg6QsRcXsrxmRmZlUqS5/G\nY5KulXQvgKSdvEa4mVnHlCVp/IZkUsHt0v1/ABPLFpGZmVWtLEmjd0T8AaiHZHZaYF1ZozIzs6qU\nJWmskLQNyUN9SBoNLCtrVGZmVpWyPBF+DjAF+KSkx4A+wNFljcrMzKpSs3caETET2A/YE/hPYOeI\neL4UlUs6RNJ8SS9L+k4j5zeX9Pv0/FOSBpWiXjMza5lm7zQkdQVOB/YmaaJ6RNKkiFhVTMWSOgNX\nAeOARcAzkqZExNy8YqeQzK77KUnHAheTTNNuZmYVkKVP40aSBZiuAK5Mt28qQd2jgJcj4pWI+Ai4\nFTiiQZkjgBvS7duAAyUJMzOriCx9GrtExE55+w9Jmttk6ez6A6/n7S8Cdm+qTLqm+DJgG+DdEtRv\nZmabKMudxsx0xBQAknYHqmoiWUkTJE2XNP2dSgdjZtaOZbnT2A14XNJr6f4OwHxJs4GIiF1bWPdi\nYPu8/QHpscbKLJLUBdgKeK/hhSJiMjAZkqnRWxiPmZk1I0vSOKRMdT8DDJa0I0lyOBb4coMyU4AT\nSaZiPxp4MJpbAMTMzMqm2aQREQvLUXHaR3EmyRQlnYHrIuIFSRcB0yNiCnAtcJOkl4GlJInFzMwq\npNmV+9oar9xnZrbpSrlyn5mZGeCkYWZmm8BJw8zMMnPSMDOzzJw0zMwsMycNMzPLzEnDzMwyc9Iw\nM7PMnDTMzCwzJw0zM8vMScPMzDJz0jAzs8ycNMzMLDMnDTMzy8xJw8zMMnPSMDOzzCqSNCT1knSf\npJfS962bKLdO0qz0NaW14zQzsw1V6k7jO8ADETEYeCDdb8zKiBiWvg5vvfDMzKwxlUoaRwA3pNs3\nAJ+vUBxmZrYJKpU0+kbEknT7TaBvE+W6Spou6UlJTixmZhXWpVwXlnQ/sG0jp76XvxMRISmauMzA\niFgs6RPAg5JmR8Q/G6lrAjABYIci4zYzs6aVLWlExNimzkl6S1K/iFgiqR/wdhPXWJy+vyJpGjAc\n2ChpRMRkYDLAyKYTkJmZFalSzVNTgBPT7ROBPzcsIGlrSZun272BvYC5rRahmZltpFJJ4yfAOEkv\nAWPTfSSNlHRNWubfgemSngMeAn4SEU4aZmYVpIj21ZozUorplQ7CzKyNEcyIiJHNlfMT4WZmlpmT\nhpmZZeakYWZmmTlpmJlZZk4aZmaWmZOGmZll5qRhZmaZOWmYmVlmThpmZpaZk4aZmWXmpGFmZpk5\naZiZWWZOGmZmlpmThpmZZeakYWZmmbW79TQkfQDMr3QcGfUG3q10EBm0lTih7cTaVuKEthNrW4kT\nqjPWgRHRp7lCZVsjvILmZ1lIpBpImt4WYm0rcULbibWtxAltJ9a2Eie0rVgbcvOUmZll5qRhZmaZ\ntcekMbnSAWyCthJrW4kT2k6sbSVOaDuxtpU4oW3FuoF21xFuZmbl0x7vNMzMrEzaVdKQdIik+ZJe\nlvSdSsfTFEnXSXpb0pxKx1KIpO0lPSRprqQXJH2j0jE1RVJXSU9Lei6N9cJKx1SIpM6SnpV0d6Vj\nKUTSq5JmS5olaXql42mKpJ6SbpM0T9KLkvaodEwNSRqS/hxzr+WSJlY6rk3VbpqnJHUG/gGMAxYB\nzwDHRcTcigbWCEn7AnXAjRGxS6XjaYqkfkC/iJgpaUtgBvD5Kv2ZCugeEXWSaoBHgW9ExJMVDq1R\nks4BRgI9IuKwSsfTFEmvAiMjotqeKdiApBuARyLiGkmbAVtExPuVjqsp6e+rxcDuEbGw0vFsivZ0\npzEKeDkiXomIj4BbgSMqHFOjIuJhYGml42hORCyJiJnp9gfAi0D/ykbVuEjUpbs16asq/yKSNAAY\nD1xT6VjaA0lbAfsC1wJExEfVnDBSBwL/bGsJA9pX0ugPvJ63v4gq/QXXFkkaBAwHnqpsJE1Lm3xm\nAW8D90VEtcZ6GfAtoL7SgWQQwN8kzZA0odLBNGFH4B3gN2mT3zWSulc6qGYcC9xS6SBaoj0lDSsT\nSbXA7cDEiFhe6XiaEhHrImIYMAAYJanqmv4kHQa8HREzKh1LRntHxAjgUOCMtGm12nQBRgC/iojh\nwAqgmvs0NwMOB/5Y6Vhaoj0ljcXA9nn7A9JjVoS0f+B24OaIuKPS8WSRNk08BBxS6VgasRdweNpX\ncCtwgKTfVjakpkXE4vT9beBOkmbgarMIWJR3Z3kbSRKpVocCMyPirUoH0hLtKWk8AwyWtGOayY8F\nplQ4pjYt7Vy+FngxIi6tdDyFSOojqWe63Y1kQMS8yka1sYj4bkQMiIhBJP+NPhgRx1c4rEZJ6p4O\ngCBt7jkIqLoRfxHxJvC6pCHpoQOBqhuskec42mjTFLSjCQsjYq2kM4GpQGfguoh4ocJhNUrSLcD+\nQG9Ji4DvR8S1lY2qUXsBXwVmp30FAOdFxD0VjKkp/YAb0lEpnYA/RERVD2dtA/oCdyZ/O9AF+F1E\n/LWyITXpv4Cb0z8YXwFOrnA8jUqT7zjgPysdS0u1myG3ZmZWfu2pecrMzMrMScPMzDJz0jAzs8yc\nNMzMLDPNQm0PAAAFMElEQVQnDTMzy8xJw6qOpJMkbZeh3PWSjs56vARxnZe3PSjLLMVpLAsknVag\nzDBJny1hnCdJurLIa0yTNDLdvif3DEwR19s/N6OvpGPSmag9JLoNctKwanQS0GzSqIDzmi/SqP+O\niEkFzg8DSpY0NpWkgs9rRcRnSzkBYET8Hji1VNez1uWkYWWV/kU+T9LN6ToHt0naIj23m6S/p5Ph\nTZXUL71DGEnyoNYsSd0knS/pGUlzJE1On1TPWv9GdaTHp0m6OF2D4x+S9kmPbyHpD0rWELlT0lOS\nRkr6CdAtjenm9PKdJV2tZP2Ov6VPojcXzxfT7/GcpIfTh9EuAo5Jr32MpFGSnkgn33s896Rzegdx\nh6S/SnpJ0iV51z05/R5PkzyUmTv+ufQ7PCvpfkl90+MXSLpJ0mPATenP+db03+hOoFveNV6V1FvS\nafp4LYgFkh5Kzx+UxjtT0h+VzFWWW99mnqSZwFFZ/82sykWEX36V7QUMIpkpda90/zrgmyRTlz8O\n9EmPH0PyFD/ANJI1HHLX6JW3fRPwuXT7euDoRuq8Hjg6Qx0/S7c/C9yfbn8T+HW6vQuwNhcLUNfg\ne60FhqX7fwCObyqWvP3ZQP90u2f6fhJwZV6ZHkCXdHsscHteuVeArYCuwEKS+db6Aa8BfYDNgMdy\n1wO25uOHeE/N+84XkKyP0i3dPyfvZ7Nrg+/9KtA7L74a4BHgc0Bv4GGStUwAvg2cn8b3OjAYUPrz\nuTvvGvvn7/vVdl7tZhoRq2qvR8Rj6fZvgbOAv5L8Ur4vvXHoDCxp4vNjJH0L2ALoBbwA3JWh3iHN\n1JGbgHEGSRIA2Bu4HCAi5kh6vsD1F0REbnqV/GsU8hhwvaQ/5NXf0FYkU6IMJkm4NXnnHoiIZQCS\n5gIDSX5xT4uId9Ljvwc+nZYfAPw+vcPaDFiQd60pEbEy3d4X+AVARDzfzPe+nGTOrLuUzNq7E/BY\n+jPeDHgCGEry83kpjem3QLVOrW6bwEnDWkPDuWqC5K/PFyKi4LKckroCvyT5q/d1SReQ/BWbRXN1\nrE7f19Gy/xdW522vI69JpykRcZqk3UkWYZohabdGiv0AeCgijlSyjsm0AnU2F/cVwKURMUXS/iR3\nGDkrmou3IUknkSSqM3OHSNYuOa5BuWGbem1rG9ynYa1hB328ZvOXSZZinQ/0yR2XVCNp57TMB8CW\n6XYuQbybtpVvyqioQnU05THgS2n5nYD/lXdujZKp4ltM0icj4qmIOJ9k4aDt2fD7QnKnkZvW/6QM\nl30K2E/SNml8X2ziWicWuMbDJP82KFmHZNdGYt+NpPnu+IjILSD1JLCXpE+lZbpL+jTJDMODJH0y\nLXdcw+tZ2+SkYa1hPskCPi+StLH/KpIleY8GLpb0HDAL2DMtfz0wScnMuquBq0mm5J5KMgV+Js3U\n0ZRfkiSaucAPSZrClqXnJgPP53WEt8RPJc1WMlz3ceA5krU/dsp1hAOXAP9X0rNkuAOKiCUkdxBP\nkCS9F/NOXwD8UdIMoNA6378CatN/o4tImtsaOpOkefChNNZr0iaxk4Bb0iatJ4ChEbGKpDnqL2lH\n+NvNfQ9rGzzLrZVV2rxyd0RU3Sp6jVEytXpNRKxK/0q+HxiSJqCWXO96ku9/WwnDbPPSprJvRsRh\nlY7FNo37NMw2tAXJX9I1JO31p7c0YaSWAT+Q1DsKP6vRYaR3U9+n8bsZq3K+0zAzs8zcp2FmZpk5\naZiZWWZOGmZmlpmThpmZZeakYWZmmTlpmJlZZv8fmXJ0FGiUch4AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x115331990>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_decision_regions(X_combined, y_combined, classifier=tree, test_idx=range(105, 150))\n",
    "plt.xlabel('petal length [standardized]')\n",
    "plt.ylabel('petal width [standardized]')\n",
    "plt.legend(loc='upper left')\n",
    "plt.show()\n",
    "# 执行上面的代码，我们得到如下结果，决策界和坐标轴平行："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "sklearn的一大优点是可以将训练好的决策树模型输出，保存在.dot文件，我们可以利用GraphViz对其可视化。\n",
    "\n",
    "先调用sklearn中export_graphviz将树模型导出:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.tree import export_graphviz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "export_graphviz(tree, out_file='tree.dot', feature_names=['petal length', 'petal width'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "\n",
    "\n",
    "然后利用GraphViz程序将tree.dot转为PNG图片:\n",
    "\n",
    "![](https://camo.githubusercontent.com/7e75a8f387daa9c408b8ac0b1df64b3705bf0ea0/68747470733a2f2f6f6f6f2e306f302e6f6f6f2f323031362f30362f32312f353736613031633436356537662e706e67)\n",
    "\n",
    "\n",
    "\n",
    "现在我们可以查看决策树在构建树时的过程：根节点105个样本，使用 petal_width <=0.75分割为两个子节点。经过第一个分割，我们可以发现左节点中样本都是同一类型，所以停止此节点的分割，右节点继续分割，注意一点，**在构建决策树时两个特征各使用了两次**。\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 随机森林\n",
    "\n",
    "随机森林一直是广受欢迎的模型，优点很多：优秀的分类表现、扩展性和使用简单。随机森林的思想也不复杂，一个随机森林模型就是多颗决策树的集成。集成学习(ensemble learning)的观点是将多个弱分类器结合来构建一个强分类器，它的泛化误差小且不易过拟合。\n",
    "\n",
    "随机森林算法大致分为4个步骤：\n",
    "* 通过自助法(bootstrap)构建大小为n的一个训练集，即重复抽样选择n个训练样例\n",
    "* 对于刚才新得到的训练集，构建一棵决策树。在每个节点执行以下操作：\n",
    "* * 通过不重复抽样选择d个特征\n",
    "* * 利用上面的d个特征，选择某种度量分割节点\n",
    "* 重复步骤1和2，k次\n",
    "* 对于每一个测试样例，对k颗决策树的预测结果进行投票。票数最多的结果就是随机森林的预测结果。至于如何投票，下面会讲到。\n",
    "\n",
    "\n",
    "随机森林中构建决策树的做法和原始决策树的区别是，在每次分割节点时，不是从所有特征中选择而是在一个小特征集中选择特征。\n",
    "\n",
    "虽然随机森林模型的可解释性不如决策树，但是它的一大优点是受超参数的影响波动不是很大(译者注：几个主要参数还是需要好好调参的)。我们也不需要对随机森林进行剪枝因为集成模型的鲁棒性很强，不会过多受单棵决策树噪音的影响。\n",
    "\n",
    "在实际运用随机森林模型时，树的数目(k)需要好好调参。一般，k越大，随机森林的性能越好，当然计算成本也越高。\n",
    "\n",
    "\n",
    "样本大小n能够控制bias-variance平衡，如果n很大，我们就减小了随机性因此随机森林就容易过拟合。另一方面，如果n很小，虽然不会过拟合，但模型的性能会降低。大多数随机森林的实现，包括sklearn中的RandomForestClassifier，n的大小等于原始训练集的大小。\n",
    "\n",
    "在每一次分割时特征集的大小d，一个最起码的要求是要小于原始特征集大小，sklearn中的默认值$d=m^{0.5}$，其中m是原始特征集大小，这是一个比较合理的数值。\n",
    "\n",
    "\n",
    "直接调用sklearn来看一下随机森林吧："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "forest = RandomForestClassifier(criterion='entropy', n_estimators=10, random_state=1, n_jobs=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RandomForestClassifier(bootstrap=True, class_weight=None, criterion='entropy',\n",
       "            max_depth=None, max_features='auto', max_leaf_nodes=None,\n",
       "            min_impurity_split=1e-07, min_samples_leaf=1,\n",
       "            min_samples_split=2, min_weight_fraction_leaf=0.0,\n",
       "            n_estimators=10, n_jobs=2, oob_score=False, random_state=1,\n",
       "            verbose=0, warm_start=False)"
      ]
     },
     "execution_count": 94,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "forest.fit(X_train, y_train) # 不需要数据标准化"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY0AAAEKCAYAAADuEgmxAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XmYVdWZ7/HvjypkEBERmiAgmGhM1KugRKOocYjGKRqH\ntOIU1G5iRyO0QzT2vQkxt28nsUOcEglR4xiNommJ0YgDDnEGFAeQSFQEHECUea567x97Fx6KqlOb\nOufUOVX1+zzPec7e+6xa661C66291tprKSIwMzPLokO5AzAzs9bDScPMzDJz0jAzs8ycNMzMLDMn\nDTMzy8xJw8zMMnPSMDOzzJw0zMwsMycNMzPLrLrcARRbLykGlTsI22DB4AHlDsHMMpj7ytyPI6J3\nU+XaXNIYBEwpdxC2wdWPX1TuEMwsg9E9R8/JUs7dU2ZmlpmThpmZZeakYWZmmbW5MY2GrNtmG+aN\nGcPqHXeEDs6Tm6itpfPs2fQfM4aOn35a7mjMrIK1i6Qxb8wYttp7bwZVV6NyB1OBAljUsyfzxoxh\nh1Gjyh2OmVWwdvFn9+odd2RbJ4xGCdi2ujq5EzMzy6NdJA06dHDCaILAXXdm1iT/ljAzs8ycNFrQ\nX599lp1PPJEdjz+en918c7nDMTPbbE4aLaSmpobzfvELHrr6ambcfTd3TprEjLffLndYZmabpV3M\nntosZ54JDU073WYbuPXWZlf74htvsOOAAXy+f38ATjnsMO5/8kl2+fznm12nmVlLc9Ko79NPkwTR\n0PUCzF+4kAF9+mw479+nDy+8/npBdZqZtTR3T5mZWWZOGi2kX+/ezP3oow3n8z76iH69m1yF2Mys\nojhptJCv7LILb733Hu/Mn8/adeu465FHOPbAA8sdlpnZZvGYRguprq7muh/8gG9ccAE1NTWcfeyx\n7PqFL5Q7LDOzzeKkUd822zQ+e6pARw0bxlHDhhVcj5lZuThp1FfAtFozs7bOYxpmZpZZ2ZKGpM6S\nXpQ0XdIbkn7SQJlOkv4oabakFyQNavlIzcysTjnvNNYAh0TEHsBg4AhJX61X5hzg04jYEfgV8PMW\njtHMzHKULWlEYnl62jF9Rb1ixwG3pMcTgEMleZVzM7MyKeuYhqQqSa8AC4BHIuKFekX6AXMBImI9\nsATYtoF6RkqaImnKwlIHbWbWjpU1aURETUQMBvoDe0varZn1jI+IoRExtJKfsT77iiv4p8MPZ7eT\nTy53KGZmzVIRs6ciYjEwGTii3kfzgQEAkqqBrYFFpY8n/3lzjTjmGP56zTXFqczMrAzKOXuqt6Qe\n6XEX4DDgzXrFJgLfSY9PAh6PKNav8IaNv68XY+/osyFRRMDYO/ow/r5eBdd94J570rN794LrMTMr\nl3LeafQFJkt6FXiJZEzjAUlXSDo2LXMjsK2k2cCFwGWlDCgClq2s4s5JPTckjrF39OHOST1ZtrKq\naHccZmatVdmeCI+IV4EhDVz/Uc7xauDbLRWTBBeelqxEe+ekntw5qScAww//hAtP+wjP2zKz9q4i\nxjQqSW7iqOOEYWaWcNKop65LKlfuGIeZWXvmpJEjdwxj+OGf8NItMxl++CcbjXEUYvh//Af7nn02\ns+bMof/RR3Pj/fcXJ3AzsxbiVW5zSLBV15qNxjDquqq26lpTcBfVnf/5n0WI0sysfJw06hl5wsdE\nsCFB1CUOj2mYmbl7qkH1E4QThplZwknDzMwyc9IwM7PMnDTMzCwzJw0zM8vMs6dayNwPP+TMMWP4\n6JNPEDDy+OMZNXx4ucMyM9ssThotpLq6ml+OHs2eX/oSy1asYK8zz+SwffZhl89/vtyhmZll5qTR\ngHe7fMLUHu+ztONquq/rzF6Lt2PQqp4F1dm3Vy/69kqWV99qyy358qBBzF+40EnDzFoVj2nU826X\nT5jc+21WVK2hU00VK6rWMLn327zb5ZPitfH++7w8axb77Lpr0eo0M2sJjd5pSDohw9evjogHixhP\n2U3t8T5VtaJjVAEk77U1TO3xfsF3GwDLV67kxEsv5aoLL6R7t24F12dm1pLydU/9DrgfyPc89IFA\ns5KGpAHArUAfIIDxEXF1vTIHpTG8k166LyKuaE57WS3tuJpONVUbXauODiztuLrgutetX8+Jl17K\naUccwQmHHFJwfWZmLS1f0ngoIs7O98WSbi+g7fXARRExTdJWwFRJj0TEjHrlno6IYwpoZ7N0X9eZ\nFVVrNtxpAKxXLd3XdS6o3ojgnJ/+lC8PGsSFp51WaJhmZmXR6JhGRJze1BdnKZPnaz+IiGnp8TJg\nJtCvufUVy16Lt6OmQ7BONQTJe02HYK/F2xVU7zPTp3Pbgw/y+JQpDD71VAafeioPPvNMkaI2M2sZ\nzR7TiIj7ihWEpEEkW7++0MDH+0qaDrwPXBwRbxSr3YYMWtWTgxey8eypRYXPntp/8GDipZeKFKWZ\nWXnk6576Zvr+T8B+wOPp+cHAs0BRkoakbsC9wOiIWFrv42nAwIhYLuko4H+AnRqoYyQwEmD7IsQ0\naFXPogx6m5m1Nfm6p86KiLOAjsAuEXFiRJwI7JpeK5ikjiQJ446G7lwiYmlELE+PHwQ6SurVQLnx\nETE0Iob2LkZgZmbWoCzPaQyIiA9yzj+iCH/QSxJwIzAzIsY2UuZzaTkk7Z3Gu6jQts3MrHmyPBH+\nmKSHgTvT85OBR4vQ9jDgDOA1Sa+k1y4nTUgRMQ44Cfg3SeuBVcApEYXu1G1mZs3VZNKIiPMlHU/y\nTAYkz1P8qdCGI+Jv5H8GhIi4Driu0LbMzKw4sq49NQ1YFhGPSuoqaat0mqyZmbUjTY5pSPpXYALw\n2/RSP5JZTJbR4mXL+M099zT766/6wx9YubrwJ9KfmDqVZ6dPL7geM2u/sgyEn0cy/rAUICLeIpmG\naxktXraM30yY0Oyvv+quu4qXNF59teB6zKz9ytI9tSYi1qaTmJBUTbJWlGV02XXX8Y/58xl86qkc\nts8+XDlqFFfedht3P/IIa9at4/iDDuIn3/0uK1at4p9/+EPmLVhATU0N/+ecc/jok094f+FCDj73\nXHr16MHkceM2rvvaa5n49NNUV1Vx+D778N+jR7Pw008597/+i/c+/BCAqy66iH69ezPu3nupqqri\n9oce4tpLLuGAIUPK8eMws1YsS9J4UtLlQBdJhwHfA/5c2rDalp+dfz6v/+MfvPKHPwAw6fnneeu9\n93jxlluICI696CKemjaNhYsXs12vXvzlqqsAWLJ8OVt368bYP/yByePG0atHj43qXbR4MX964gne\nnDABSSxelgwzjfrlL/n3U09l/8GDee/DD/nG97/PzHvu4dwTT6Rbly5cfMYZLfsDMLM2I0vSuAw4\nB3gN+C7wYET8rqRRtXGTnn+eSS+8wJB04cLlq1bx1ty5HDB4MBdddRWXXnstx+y/f5N3Alt360bn\nTp0456c/5Zj99+eYAw4A4NEXX2TG229vKLd0xQqWr1xZum/IzNqNLEnjNOCu3EQh6ZiIeKB0YbVt\nEcEPR4zguydsurzXtNtu48FnnuF/X389h37lK/zoX/+10Xqqq6t58eabeeyll5jw2GNcd889PH79\n9dTW1vL8739P506dSvltmFk7lGUg/FrgaUlfzrlW0j0t2pqtunZlWc5f+t/Yd19umjhxw1//8xcs\nYEE6dtG1c2dOP+ooLjnjDKbNmvXZ169YsUm9y1euZMny5Rw1bBi/uvBCpr/1FgCHf/WrXHv33RvK\nvZJbj+84zKwAWe403iHpnpogaUxE3EMTD+XZxrbt0YNhe+zBbiefzJH77ceVo0Yx85132PfsZLuS\nbl27cvsVVzB77lwuueYaOkh0rK7m+ssuA2Dk8cdzxAUXsF3v3hsNhC9buZLjLrqI1WvXEhGMHT0a\ngGsuvpjzfv5zdh8+nPU1NRw4ZAjjfvhDvnnAAZx02WXc/+STHgg3s2ZRU6tySJoWEXumCwXeCUwH\nDo+I3VsiwM01VIop9a7NfOghvtxrk3UOrZ6ZH3/Ml488sqh1Xv3JVUWtz8xKY3TP0VMjYmhT5bJ0\nT30AEBEfA98gmW67W2HhmZlZa9Rk0oiIo3OOayPikojIkmzMzKyNybdz31URMVrSn2ngYb6IOLak\nkRVTbS2BB2LyCYDa2nKHYWYVLt9A+G3p+3+3RCCl1Hn2bBb17Mm21dVOHA0IYNH69XSePbvcoZhZ\nhWs0aUTE1PT9yZYLpzT6jxnDvDFjWLjjjtDBPWubqK2l8+zZ9B8zptyRmFmFy9c99Rp51piq1NlT\nDen46afsMGpUucMwM2v18nVPHZO+n5e+13VXnU4RFiyUNAC4FeiT1jc+Iq6uV0bA1cBRwEpgRERM\nK7RtMzNrnnzdU3MAJB0WEblPgV0qaRrJmlSFWA9cFBHTJG0FTJX0SETMyClzJLBT+toHuD59NzOz\nMsjSwS9Jw3JO9sv4dXlFxAd1dw3pLoAzSTZ4ynUccGskngd6SOpbaNtmZtY8WZYRORv4vaSt0/PF\n6bWikTQIGAK8UO+jfsDcnPN56bUPitm+mZllkzdpSOoA7BgRe9QljYhYUswAJHUD7gVGR8TSZtYx\nEhgJsH0RYzMzs43l7WaKiFrgB+nxkhIkjI4kCeOOiLivgSLzgQE55/3Ta/XjHB8RQyNiaO9iBmhm\nZhvJMjbxqKSLJQ2Q1LPuVWjD6cyoG4GZETG2kWITgTOV+CqwJCLcNWVmViZZxjROTt/Py7kWwOcL\nbHsYcAbwmqRX0muXk/YwRcQ44EGS6bazSabcnlVgm2ZmVoAmk0ZE7FCKhiPibzSxHFQk67afl6+M\nmZm1nCx3GkjaDdgF6Fx3LSJuLVVQZmZWmZpMGpJ+DBxEkjQeJHng7m8kT3ObWSsUAVLj52aNyTIQ\nfhJwKPBhRJwF7AFsnf9LzKxSPXvzbrzzfB/qNu2MgHee78OcqZ57aE3L0j21KiJqJa2X1B1YwMbT\nYM2slYiAz+28kIFfWUBtDXSogtoaGLT3R0i+47CmZUkaUyT1AH4HTAWWA8+VNCozKwkJdvjqR9TW\nQO36KmrXJ9erO9Vs+Nwsnyyzp76XHo6T9Fege0S8WtqwzKxUpPQOY/2m182akm8/jT3zfeYlys1a\np4ikS6qh6/kGx911ZZD/TuOX6XtnYCgwneS5it2BKcC+pQ3NzIqtbtB74FcW0KG6ZsOYRtQmCSFI\n3mvWpwmiAxvGOlYv3YII6Npjbbm/DSujfPtpHAwg6T5gz4h4LT3fDRjTItGZWVFJcM3Rp/D1C1/g\n4PNepnO3taxevgWTf51smVP/2qNjk+1rjrviKRbN3YpDz59Gl63X+o6jHcsyEL5zXcIAiIjXJX25\nhDGZWYk9OnafDQmh/vWGTLtvZ/Y8YRY9+i93wmjnsiSNVyXdANyenp8GeCDcrJ056LyXnTAs08N9\nZwFvAKPS1wy8cKBZRal7UC/reXNMvm7IRvUUo05rfZpMGhGxOiJ+FRHHp69fRcTqlgjOzJo2Y9LA\nDYPU8Nmg9YxJA4HkCfAnfj2koF/yPbdfwpO/HbIhcUTAE78ewrM371aE78BakyxrTw0jGfgemFs+\nIgpdGt3agdE9R5U7hHbhS4e8yyEXTGXbgUtZNKc7j1+zF28+PmijMvf/6MBm1z9w6Ie8O6Uvc6b2\nBV7miV8P2TDO4am47UuWMY0bgX8neRq8gdndZlZubz4+aJMkUUwHn/cyAC/ftzNjDz0FgD1PmOVx\njnYoy5jGkoh4KCIWRMSiulfJIzOziiF9ljjqOGG0T1nuNCZLuhK4D1hTd7EYT4RLugk4BlgQEZt0\njko6CLgfeCe9dF9EXFFou2bWtLEfX02H9M/KujGMXE/8eogTRzuUJWnUTdwemnMtgEOK0P7NwHXk\n35vj6Yg4pghtmVlGYz++mod/sTeDv/UWn9v5UxbP60Z1l3WcfcsD9Oi/nMXzuvHYdXs6cbRDWRYs\nPLhUjUfEU5IGlap+M2ueh3+xN3sPf5MOVckwZpdtVjFsxOsoXVakR//lHH3587zzQl8njHYm63av\nRwO7svF2ry3VTbSvpOnA+8DFEfFGC7Vr1m7ttP98OlTV0KPfCiTo3K1mo1lSEnTuvpZdDp9T3kCt\nxWWZcjsO6AocDNxAspPfiyWOq840YGBELJd0FPA/wE4NxDgSGAmwfQsFZtaWbbP90g0Jo079Owrf\nYbRPWWZP7RcRZwKfRsRPSFa3/WJpw0pExNKIWJ4ePwh0lNSrgXLjI2JoRAz1hpVmhfv0ve4snr+l\nn/q2TWRJGqvS95WStgPWAX1LF9JnJH1OSv6ekbQ3Sbye7mtWYm/9rR+1NVVOHLaJLGMaD6TbvV5J\n0l0UJN1UBZN0J3AQ0EvSPODHQEeAiBhH0hX2b5LWkySvUyL8n7BZqT38i32ZM6Uvh1wwldqaKhbN\n6c7fn+rPFw+cl/ep883xpUPe5aQrJ7N+TRVrV1WzRZf1VHeqYcIlB5f0QUVrzOhMpdTU72BJnSJi\nTd0xyWD46rprlWaoFFPKHYRtIJzjrWHf+5976f5PK1i7quOGa1t0WcfSBVvym2+dWMbI2itNjYih\nTZXK0j31XN1BRKyJiCW518zMmmPbgUtZu2rjzo61q6rZduDSMkVkWeTbI/xzQD+gi6QhJFu9AnQn\nmU1lZtZsi+Z0b+BOYz2L5nQvY1TWlHxjGt8ARgD9SfYLr0say4DLSxuWmbV1j1+zFyddORlgozGN\nx6/Zq8yRWT5ZxjROjIh7WyiegnlMo7J4TMPyybKku7WUbGMaWWZP9ZfUneQO43fAnsBlETGpwAjN\nrJ0r9ZLuVnxZBsLPjoilwOHAtsAZwM9KGpWZmVWkLEmjbizjKODWdO0nLyBgZtYOZUkaUyVNIkka\nD0vaCqgtbVhmZlaJsoxpnAMMBt6OiJWStgXOKm1YZmZWiRq900if0yAiaiNiWkQsTs8XRcSruWXM\nzKx9yNc99WCGr89SxszM2oh83VN7SMr3PL8AP+9vZtaONJo0IqKqJQMxM7PKl2X2lJmZGeCkYWZm\nm8FJw8zMMsuUNCRVSdpO0vZ1r2I0LukmSQskvd7I55J0jaTZkl6VtGcx2jUzs+ZpMmlI+j7wEfAI\n8Jf09UCR2r8ZOCLP50cCO6WvkcD1RWrXzMyaIcsT4aOAnSNiUbEbj4inJA3KU+Q4kvWuAnheUg9J\nfSPig2LHYmZmTcvSPTUXWFLqQBrRL22/zrz02kYkjZQ0RdKUhS0WmplZ+5Nvu9cL08O3gSck/QVY\nU/d5RIwtcWyZRcR4YDwkmzCVORwzszYrX/fUVun7e+lri/QFtNh2bPOBATnn/dNrZmZWBvmeCP8J\ngKRvR8Q9uZ9J+napA0tNBM6XdBewD7DE4xlmZuWTZUzjhxmvbTZJdwLPATtLmifpHEnnSjo3LfIg\nSffYbJKtZr9XjHbNzKx58o1pHEmy8VI/SdfkfNQdWF+MxiNieBOfB3BeMdoyM7PC5RvTeB+YChyb\nvtdZBvx7KYMyM7PKlG9MYzowXdIdEbGuBWMyM7MKla976jXSWVKSNvk8InYvXVhmZlaJ8nVPHZO+\n140p3Ja+n07LTbk1M7MKomSsOU8B6eWIGFLv2rSIqMjFA4dKMaXcQdgG8t8XZq2EpkbE0KZKZZly\nK0nDck72y/h1ZmbWxmRZsPAc4CZJW5PsC/4pcHZJozIzs4rUZNKIiKnAHmnSICLKtXihmZmVWb7Z\nU6dHxO05CxfWXQcqa8FCMzNrGfnuNLZM37fKU8Ysr2DT6dp1PEhu1vrke7jvt+nhzyNidQvFY2Zm\nFSzLQPjrkj4Cnk5ff/O4hplZ+9Tk1NmI2BEYDrwGHE2ytMgrpQ7MzMwqT5N3GpL6A8OAA4A9gDeA\nv5U4LjMzq0BZuqfeA14C/l9EnNtUYTMza7uyPNk9BLgVOFXSc5JulXROieMyM7MKlGVMYzpwC/B7\n4HHga8CPitG4pCMkzZI0W9JlDXw+QtJCSa+kr38pRrtmZtY8WcY0pgCdgGdJZk8dGBFzCm1YUhXw\na+AwYB7wkqSJETGjXtE/RsT5hbZnZmaFyzKmcWRELCxB23sDsyPibQBJdwHHAfWThpmZVYgs3VOl\nSBgA/YC5Oefz0mv1nSjpVUkTJA1oqCJJIyVNkTSlVMGamVnlL3H+Z2BQukvgIyRjK5uIiPERMTQi\nhvZu0fDMzNqXciaN+UDunUP/9NoGEbEoItakpzcAe7VQbGZm1oB8q9yekO8LI+K+Att+CdhJ0g4k\nyeIU4NR6MfSNiA/S02OBmQW2aWZmBcg3EP7NPJ8FUFDSiIj1ks4HHgaqgJsi4g1JVwBTImIicIGk\nY4H1wCfAiELaNDOzwjS5R3hr4z3CWw8vjW5WSbLtEZ5lyi2SjgZ2BTrXXYuIK5ofnJmZtUZNDoRL\nGgecDHyfZI/wbwMDSxyXmZlVoCyzp/aLiDOBTyPiJ8C+wBdLG5aZmVWiLEljVfq+UtJ2wDqgb+lC\nMjOzSpVlTOMBST2AK4FpJDOnbihpVGZmVpGanD0lqVPdA3aSOpEMhq/Oeeiuonj2VOvh2VNmlSTb\n7Kks3VPP1R1ExJp0f/Dn8pQ3M7M2Kt8T4Z8jWUCwi6QhJDOnALoDXVsgNjMzqzD5xjS+QfIEdn9g\nbM71pcDlJYzJzMwqVKNJIyJuAW6RdGJE3NuCMZmZWYXKMqbxjKQbJT0EIGkX7xFuZtY+ZUkavydZ\nVHC79PzvwOiSRWRmZhUrS9LoFRF3A7WQrE4L1JQ0KjMzq0hZksYKSduSPNSHpK8CS0oalZmZVaQs\nT4RfCEwEviDpGaA3cFJJozIzs4rU5J1GREwDvgbsB3wX2DUiXi1G45KOkDRL0mxJlzXweSdJf0w/\nf0HSoGK0a2ZmzdPknYakzsD3gP1JuqieljQuIlYX0rCkKuDXwGHAPOAlSRMjYkZOsXNIVtfdUdIp\nwM9Jlmk3M7MyyDKmcSvJBkzXAtelx7cVoe29gdkR8XZErAXuAo6rV+Y44Jb0eAJwqCRhZmZlkWVM\nY7eI2CXnfLKkGY2Wzq4fMDfnfB6wT2Nl0j3FlwDbAh8XoX0zM9tMWe40pqUzpgCQtA9QUQvJShop\naYqkKQvLHYyZWRuW5U5jL+BZSe+l59sDsyS9BkRE7N7MtucDA3LO+6fXGiozT1I1sDWwqH5FETEe\nGA/J0ujNjMfMzJqQJWkcUaK2XwJ2krQDSXI4BTi1XpmJwHdIlmI/CXg8mtoAxMzMSqbJpBERc0rR\ncDpGcT7JEiVVwE0R8YakK4ApETERuBG4TdJs4BOSxGJmZmXS5M59rY137ms9vHOfWSUp3s59ZmZm\ngJOGmZltBicNMzPLzEnDzMwyc9IwM7PMnDTMzCwzJw0zM8vMScPMzDJz0jAzs8ycNMzMLDMnDTMz\ny8xJw8zMMnPSMDOzzJw0zMwsMycNMzPLzEnDzMwyK0vSkNRT0iOS3krft2mkXI2kV9LXxJaO08zM\nNlauO43LgMciYifgsfS8IasiYnD6OrblwjMzs4aUK2kcB9ySHt8CfKtMcZiZ2WYoV9LoExEfpMcf\nAn0aKddZ0hRJz0tyYjEzK7PqUlUs6VHgcw189B+5JxERkqKRagZGxHxJnwcel/RaRPyjgbZGAiMB\nti8wbjMza1zJkkZEfL2xzyR9JKlvRHwgqS+woJE65qfvb0t6AhgCbJI0ImI8MB5gaOMJyMzMClSu\n7qmJwHfS4+8A99cvIGkbSZ3S417AMGBGi0VoZmabKFfS+BlwmKS3gK+n50gaKumGtMyXgSmSpgOT\ngZ9FhJOGmVkZKaJt9eYMlWJKuYOwTETb+m/PrHXT1IgY2lQpPxFuZmaZOWmYmVlmThpmZpaZk4aZ\nmWXmpGFmZpk5aZiZWWZOGmZmlpmThpmZZeakYWZmmTlpmJlZZk4aZmaWmZOGmZll5qRhZmaZOWmY\nmVlmThpmZpZZm9tPQ9IyYFa548ioF/BxuYPIoLXECa0n1tYSJ7SeWFtLnFCZsQ6MiN5NFSrZHuFl\nNCvLRiKVQNKU1hBra4kTWk+srSVOaD2xtpY4oXXFWp+7p8zMLDMnDTMzy6wtJo3x5Q5gM7SWWFtL\nnNB6Ym0tcULribW1xAmtK9aNtLmBcDMzK522eKdhZmYl0qaShqQjJM2SNFvSZeWOpzGSbpK0QNLr\n5Y4lH0kDJE2WNEPSG5JGlTumxkjqLOlFSdPTWH9S7pjykVQl6WVJD5Q7lnwkvSvpNUmvSJpS7nga\nI6mHpAmS3pQ0U9K+5Y6pPkk7pz/HutdSSaPLHdfmajPdU5KqgL8DhwHzgJeA4RExo6yBNUDSgcBy\n4NaI2K3c8TRGUl+gb0RMk7QVMBX4VoX+TAVsGRHLJXUE/gaMiojnyxxagyRdCAwFukfEMeWOpzGS\n3gWGRkSlPVOwEUm3AE9HxA2StgC6RsTicsfVmPT31Xxgn4iYU+54NkdbutPYG5gdEW9HxFrgLuC4\nMsfUoIh4Cvik3HE0JSI+iIhp6fEyYCbQr7xRNSwSy9PTjumrIv8iktQfOBq4odyxtAWStgYOBG4E\niIi1lZwwUocC/2htCQPaVtLoB8zNOZ9Hhf6Ca40kDQKGAC+UN5LGpV0+rwALgEciolJjvQr4AVBb\n7kAyCGCSpKmSRpY7mEbsACwEfp92+d0gactyB9WEU4A7yx1Ec7SlpGElIqkbcC8wOiKWljuexkRE\nTUQMBvoDe0uquK4/SccACyJiarljyWj/iNgTOBI4L+1arTTVwJ7A9RExBFgBVPKY5hbAscA95Y6l\nOdpS0pgPDMg5759eswKk4wP3AndExH3ljieLtGtiMnBEuWNpwDDg2HSs4C7gEEm3lzekxkXE/PR9\nAfAnkm7gSjMPmJdzZzmBJIlUqiOBaRHxUbkDaY62lDReAnaStEOayU8BJpY5plYtHVy+EZgZEWPL\nHU8+knpL6pEedyGZEPFmeaPaVET8MCL6R8Qgkv9GH4+I08scVoMkbZlOgCDt7jkcqLgZfxHxITBX\n0s7ppUOBipuskWM4rbRrCtrQgoURsV7S+cDDQBVwU0S8UeawGiTpTuAgoJekecCPI+LG8kbVoGHA\nGcBr6VjoZQEAAAAFlElEQVQBwOUR8WAZY2pMX+CWdFZKB+DuiKjo6aytQB/gT8nfDlQDf4iIv5Y3\npEZ9H7gj/YPxbeCsMsfToDT5HgZ8t9yxNFebmXJrZmal15a6p8zMrMScNMzMLDMnDTMzy8xJw8zM\nMnPSMDOzzJw0rOJIGiFpuwzlbpZ0UtbrRYjr8pzjQVlWKU5jeUfSuXnKDJZ0VBHjHCHpugLreELS\n0PT4wbpnYAqo76C6FX0lnZyuRO0p0a2Qk4ZVohFAk0mjDC5vukiDLomIcXk+HwwULWlsLkl5n9eK\niKOKuQBgRPwR+Jdi1Wcty0nDSir9i/xNSXek+xxMkNQ1/WwvSU+mi+E9LKlveocwlORBrVckdZH0\nI0kvSXpd0vj0SfWs7W/SRnr9CUk/T/fg+LukA9LrXSXdrWQPkT9JekHSUEk/A7qkMd2RVl8l6XdK\n9u+YlD6J3lQ8306/j+mSnkofRrsCODmt+2RJe0t6Ll1879m6J53TO4j7JP1V0luSfpFT71np9/Ei\nyUOZdde/mX4PL0t6VFKf9PoYSbdJega4Lf0535X+G/0J6JJTx7uSekk6V5/tBfGOpMnp54en8U6T\ndI+Stcrq9rd5U9I04ISs/2ZW4SLCL79K9gIGkayUOiw9vwm4mGTp8meB3un1k0me4gd4gmQPh7o6\neuYc3wZ8Mz2+GTipgTZvBk7K0MYv0+OjgEfT44uB36bHuwHr62IBltf7vtYDg9Pzu4HTG4sl5/w1\noF963CN9HwFcl1OmO1CdHn8duDen3NvA1kBnYA7Jemt9gfeA3sAWwDN19QHb8NlDvP+S8z2PIdkf\npUt6fmHOz2b3et/3u0CvnPg6Ak8D3wR6AU+R7GUCcCnwozS+ucBOgNKfzwM5dRyUe+5X63m1mWVE\nrKLNjYhn0uPbgQuAv5L8Un4kvXGoAj5o5OsPlvQDoCvQE3gD+HOGdnduoo26BRinkiQBgP2BqwEi\n4nVJr+ap/52IqFteJbeOfJ4BbpZ0d0779W1NsiTKTiQJt2POZ49FxBIASTOAgSS/uJ+IiIXp9T8C\nX0zL9wf+mN5hbQG8k1PXxIhYlR4fCFwDEBGvNvF9X02yZtaflazauwvwTPoz3gJ4DvgSyc/nrTSm\n24FKXVrdNoOThrWE+mvVBMlfn29ERN5tOSV1Bn5D8lfvXEljSP6KzaKpNtak7zU07/+FNTnHNeR0\n6TQmIs6VtA/JJkxTJe3VQLGfApMj4ngl+5g8kafNpuK+FhgbERMlHURyh1FnRVPx1idpBEmiOr/u\nEsneJcPrlRu8uXVb6+AxDWsJ2+uzPZtPJdmKdRbQu+66pI6Sdk3LLAO2So/rEsTHaV/55syKytdG\nY54B/jktvwvwv3I+W6dkqfhmk/SFiHghIn5EsnHQADb+fiG506hb1n9EhmpfAL4mads0vm83Utd3\n8tTxFMm/DUr2Idm9gdj3Ium+Oz0i6jaQeh4YJmnHtMyWkr5IssLwIElfSMsNr1+ftU5OGtYSZpFs\n4DOTpI/9+ki25D0J+Lmk6cArwH5p+ZuBcUpW1l0D/I5kSe6HSZbAz6SJNhrzG5JEMwP4vyRdYUvS\nz8YDr+YMhDfHlZJeUzJd91lgOsneH7vUDYQDvwD+S9LLZLgDiogPSO4gniNJejNzPh4D3CNpKpBv\nn+/rgW7pv9EVJN1t9Z1P0j04OY31hrRLbARwZ9ql9RzwpYhYTdId9Zd0IHxBU9+HtQ5e5dZKKu1e\neSAiKm4XvYYoWVq9Y0SsTv9KfhTYOU1AzanvZpLvf0IRw2z10q6yiyPimHLHYpvHYxpmG+tK8pd0\nR5L++u81N2GklgA/ldQr8j+r0W6kd1M/puG7GatwvtMwM7PMPKZhZmaZOWmYmVlmThpmZpaZk4aZ\nmWXmpGFmZpk5aZiZWWb/H6uvpz9h18lQAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1145c0310>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_decision_regions(X_combined, y_combined, classifier=forest, test_idx=range(105, 150))\n",
    "plt.xlabel('petal length [standardized]')\n",
    "plt.ylabel('petal width [standardized]')\n",
    "plt.legend(loc='upper left')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "运行上面的代码，我们训练了一个含有10颗树的随机森林，使用熵作为分割节点时的度量。虽然我们在一个小数据集上训练了一个非常小的模型，但我还是使用了n_jobs这个并行化参数，此处使用了计算机的两个核训练模型。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## k近邻--一个懒惰学习算法\n",
    "\n",
    "本章我们要讨论的最后一个监督学习算法是k紧邻算法(k-nearest neighbor classifier, KNN), 这个算法很有意思，因为他背后的思想和本章其他算法完全不同。\n",
    "\n",
    "\n",
    "KNN是懒惰学习的一个典型示例。之所以称为“懒惰”并不是由于此类算法看起来很简单，而是在训练模型过程中这类算法并不去学习一个判别式函数(损失函数)而是要记住整个训练集。\n",
    "\n",
    "\n",
    "**Note** 参数模型VS变参模型\n",
    "\n",
    "机器学习算法可以被分为两大类：参数模型和变参模型。对于参数模型，在训练过程中我们要学习一个函数，重点是估计函数的参数，然后对于新数据集，我们直接用学习到的函数对齐分类。典型的参数模型包括感知机、逻辑斯蒂回归和线性SVM。与之相对的，变参模型中的参数个数不是固定的，它的参数个数随着训练集增大而增多！很多书中变参(nonparametric)被翻译为无参模型，一定要记住，不是没有参数，而是参数个数是变量！变参模型的两个典型示例是决策树/随机森林和核SVM。\n",
    "\n",
    "KNN属于变参模型的一个子类：基于实例的学习(instance-based learning)。基于实例的学习的模型在训练过程中要做的是记住整个训练集，而懒惰学习是基于实例的学习的特例，在整个学习过程中不涉及损失函数的概念。\n",
    "\n",
    "\n",
    "KNN算法本身非常简单，步骤如下：\n",
    "* 1 确定k大小和距离度量。\n",
    "* 2 对于测试集中的一个样本，找到训练集中和它最近的k个样本。\n",
    "* 3 将这k个样本的投票结果作为测试样本的类别。\n",
    "\n",
    "一图胜千言，请看下图：\n",
    "\n",
    "\n",
    "\n",
    "![](https://ooo.0o0.ooo/2016/06/22/576a3afd1bb03.png)\n",
    "\n",
    "\n",
    "\n",
    "对每一个测试样本，基于事先选择的距离度量，KNN算法在训练集中找到距离最近(最相似)的k个样本，然后将k个样本的类别的投票结果作为测试样本的类别。\n",
    "\n",
    "\n",
    "像KNN这种基于内存的方法一大优点是：一旦训练集增加了新数据，模型能立刻改变。另一方面，缺点是分类时的最坏计算复杂度随着训练集增大而线性增加，除非特征维度非常低并且算法用诸如KD-树等数据结构实现。此外，我们要一直保存着训练集，不像参数模型训练好模型后，可以丢弃训练集。因此，存储空间也成为了KNN处理大数据的一个瓶颈。\n",
    "\n",
    "\n",
    "下面我们调用sklearn训练一个KNN模型："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.neighbors import KNeighborsClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "knn = KNeighborsClassifier(n_neighbors=5, p=2, metric='minkowski')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',\n",
       "           metric_params=None, n_jobs=1, n_neighbors=5, p=2,\n",
       "           weights='uniform')"
      ]
     },
     "execution_count": 98,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "knn.fit(X_train_std, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY0AAAEKCAYAAADuEgmxAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl8VPW9//HXJwtrAA1QZBGxreJVryVIEcUFFFErtdXa\nq1RrcSn1qrWovW793Yq2tWqvVutSihtqrdi6tJZqFVBccGVxQSWuyC4gCgRZkszn98c5EybDZHKS\nzGRmwvv5eMxjzjbnfBI0nznfz/l+v+buiIiIRFGU6wBERKRwKGmIiEhkShoiIhKZkoaIiESmpCEi\nIpEpaYiISGRKGiIiEpmShoiIRKakISIikZXkOoBMK+te5uX9y3MdhohIQVny+pI17t6zsePaXNIo\n71/ORU9flOswREQKyoTyCZ9EOU7NUyIiEpmShoiIRKakISIikbW5mkYqpbWlDFw7kLLqslyHkreq\nSquoLK+kurg616GISB7bIZLGwLUD2W2n3SjbuQwzy3U4ecfdqVpbBWthQc8FuQ5HRPLYDtE8VVZd\npoSRhplRVl6mOzERadQOkTQAJYxG6PcjIlHsMElDRERaTkmjFT395NMM32c4w/Yaxs3X3ZzrcERE\nmkxJo5XU1tZy2fmX8Zd//oXn3nyOR6c+SuU7lbkOS0SkSXaIp6eaonzYURSt/my77bGe3Vn78pPN\nPu/8V+ez+9d2Z7ev7gbAd0/6Lk/+80kG7j2w2ecUEWltShpJilZ/RqzH9gMepkokTbFi+Qr69OtT\nt967b2/mvTqvRecUEWltap4SEZHIlDRaSe8+vVm+dHnd+oplK+jdt3cOIxIRaToljVYy6JuD+OiD\nj/jk40/YunUrf3/w74weMzrXYYmINIlqGq2kpKSEq2+6mrHHjqW2tpax48ay1z575TosEZEmUdJI\nEuvZvcGnp1pq1DGjGHXMqBafR0QkV5Q0krTksVoRkbZONQ0REYksZ0nDzDqY2atm9oaZvW1mV6Y4\npr2ZPWhmH5jZK2Y2oPUjFRGRuFzeaWwBDnf3bwCDgKPNbFjSMWcCn7v714HfA9e2cowiIpIgZ0nD\nA1Xhamn48qTDvgPcEy4/BBxhGsNbRCRnclrTMLNiM3sdWAVMd/dXkg7pCywBcPcaYB2w3WNMZjbe\nzOaY2ZyqNVXJu0VEJENymjTcvdbdBwH9gKFmtm8zzzPZ3Ye4+5CyHvk7+9yEsyawT599OGzQYbkO\nRUSkWfLi6Sl3/wJ4Bjg6adcyYFcAMysBugEtGzkwUjzp15vrpB+dxAPTHsjMyUREciCXT0/1NLOd\nwuWOwJHAwqTDHgN+FC6fCDztnqk/4andd3tHJt3YqS5RuMOkGztx3+0dW3zuAw85kJ3Kd2rxeURE\nciWXdxq9gWfM7E3gNYKaxjQzu8rMjguPuRPobmYfABcCl2YzIHeoqjIendqhLnFMurETj07tQFWV\nZeyOQ0SkUOWsR7i7vwlUpNj+y4TlzcD3WysmMzh7wpcAPDq1A49O7QDA8Sdv5uwJX6LntkRkR5cX\nNY18kpg44pQwREQCShpJ4k1SiRJrHCIiOzIljQSJNYzjT97Mky+v5fiTN9ercbTE2aeezZhDxvBh\n5YdUDKjgL3f9JTOBi4i0Eo1ym8AMysq8Xg0j3lRVVuYtbqKa9OdJGYhSRCR3lDSS/PDHm3CnLkHE\nE4dqGiIiap5KKTlBKGGIiASUNEREJDIlDRERiUxJQ0REIlPSEBGRyJQ0WsmyJcs4YdQJHLLfIRz6\njUO5/Q+35zokEZEm0yO3raSkpISJ101kv8H7UbWhitEHjObQUYcycO+BuQ5NRCQyJY0UllcvZ2H1\nQjbGNtK5qDN7le5Fn9I+LTpnr9696NW7FwBlXcrYY689WLl8pZKGiBQUJY0ky6uXM3fLXIooopRS\nNsU2MXfLXIAWJ464xYsWs+D1BQweOjgj5xMRaS0NJg0zOyHC5ze7++MZjCfnFlYvpIgiSiz41ZRQ\nQo3XsLB6YUaSxsaqjZz1X2dx1fVX0aVrlxafT0SkNaW707gd+AeQrj/0oUCzkoaZ7QrcC/QCHJjs\n7jclHTMijOHjcNMj7n5Vc64X1cbYRkoprbetmGI2xja2+NzV1dWc+V9ncsLYEzj2+GNbfD4RkdaW\nLmk84e5npPuwmf25BdeuAS5y93lm1gWYa2bT3f2dpOOed/cxLbhOk3Qu6sym2CZKEn41tdTSuahz\ni87r7lzw4wvYY689OPuCs1sapohITjT4yK27n9rYh6Mck+azK9x9Xri8AXgX6Nvc82XKXqV7ESNG\njdfg7tR4DTFi7FW6V4vO++rsV3no/od44ZkXOGL/Izhi/yOY8cSMDEUtItI6ml3TcPdHMhWEmQ0g\nmPr1lRS7DzSzN4DlwM/d/e1MXTeVeN0i009PHXDwAaysXpmJEEVEciZd89S3w/evAAcBT4frI4EX\ngYwkDTMrAx4GJrj7+qTd84Dd3L3KzL4F/B3YI8U5xgPjAXbut3OLY+pT2idjT0qJiLQl6ZqnTnf3\n04FSYG93/567fw/YJ9zWYmZWSpAw7k915+Lu6929Klx+HCg1sx4pjpvs7kPcfUhZj7JMhCYiIilE\nGUZkV3dfkbD+KdC/pRc2MwPuBN519xsaOGaX8DjMbGgY72ctvbaIiDRPlM59M83sSeCBcP0kIBMV\n3OHAD4G3zOz1cNvlhAnJ3ScBJwL/bWY1wCbgZPeWztQtIiLN1WjScPfzzOx4gj4ZEPSneLSlF3b3\nF0jfBwR3vwW4paXXEhGRzIg6jMg8YIO7zzCzTmbWJXxMVkREdiCN1jTM7MfAQ8Cfwk19CZ5ikojW\nfbGOu/94d7M/P/mmyXz55ZctjmP2s7N57cXXWnweEdlxRSmEn0tQf1gP4O7vEzyGKxGt+2IdU/40\npdmfn3zzZDZ9uanFcbz47Iu89rKShog0X5SkscXdt8ZXzKyEYKwoieg3v/gNn3z4CUfsfwRXXnIl\nALdefytHDTuKkRUjue7K6wDYuHEjpxx3CocPPpzDBh3G3//6d+64+Q4+Xf4p3xv1PU4YtX1/y19f\n/msO2e8QRlaMZOLFEwFYs3oNZ/7XmRw17CiOGnYUr85+lcWLFnPv5HuZfNNkjtj/CF5+4eVW+/lF\npO2IUtN41swuBzqa2ZHAOcA/sxtW2/KL3/yChW8vZObcmQDMmj6Lj9//mH+/9G/cndOOP42Xnn+J\nz1Z/xi69d+H+x+4HYP269XTt1pVJN03i4RkP071H93rnXfvZWp74xxO8sOAFzIx1X6wD4H8v/F/G\nnz+eAw4+gKWLlzL22LE8/9bznDb+NDqXdeacC89p3V+AiLQZUZLGpcCZwFvAT4DH3V1zlbbArOmz\nmDVjFqOGjAKCO4yP3/+YAw4+gIkXT+RXl/2KI489kmEHD0t7nq7dutK+fXsu+PEFHHnskRx57JEA\nPDfzOd5757264zas38DGqpaP0isiEiVpnAJMTUwUZjbG3adlL6y2zd05/+LzOW38advtm/7qdGY+\nMZNrfnkNhxx+CBf9v4saPE9JSQlPvPQEzz/9PNMensZdt93Fw9MfxmPOv2b/iw4dOmTzxxCRHVCU\nmsbNwPNm9h8J27I6p0VbU9aljKoNVXXrI0eP5IEpD9R9+1+xbAWrV61m5fKVdOzUkRNPOZFzLjqH\nt+a/FXy+rP7n4zZWbWT9uvWMOmYUV11/Fe+8GYwqf9iRh3HnrXfWHbfg9QUp4xARaaoodxofEzRP\nPWRmE939bzTSKU/qK+9eztCDhnLYoMM4/KjDueLaK3j/3fc59uBgIqbOZZ259Z5b+fjDj7nqkqso\nKiqipLSEa2+5FoBTzzqVH4z5Ab169+KRGduG6KraUMWPTvgRWzZvwd2Z+LuJAPz697/msvMvY2TF\nSGpqazjw4AO57rbrGH3saM466SyefOxJfnPTbxpt/hIRSWaNjcphZvPcfXA4UOADwBvAaHffrzUC\nbKr+Ff39oqfrN+kMWz6MAQMH5CagArKochEv99FTVSI7ognlE+a6+5DGjovSPLUCwN3XAEcRPG67\nb8vCExGRQtRo0nD3YxOWY+7+P+4eJdmIiEgbk27mvhvdfYKZ/ZMUnfnc/bisRpZh7k44yrqkoMGD\nRSSKdIXw+8L3/2uNQLKpqrSKqrVVlJWXKXGk4O5Ura2iqlRPVolIeg0mDXefG74/23rhZEdleSWs\nhbI1mtWvIVWlVcHvSUQkjXTNU2+RZoypfH16KpXq4moW9FyQ6zBERApeuuapMeH7ueF7vLnqVDIw\nYKGZ7QrcC/QKzzfZ3W9KOsaAm4BvAV8C49x9XkuvLSIizZOueeoTADM70t0rEnZdYmbzCMakaoka\n4CJ3n2dmXYC5Zjbd3d9JOOYYYI/wdQDwx/BdRERyIMqjs2ZmwxNWDor4ubTcfUX8riGcBfBdggme\nEn0HuNcDLwM7mVnvll5bRESaJ8owImcAd5tZt3D9i3BbxpjZAKACeCVpV19gScL60nDbikxeX0RE\nokmbNMysCPi6u38jnjTcfV0mAzCzMuBhYIK7r2/mOcYD4wF27rdzBqMTEZFEaZuZ3D0GXBwur8tC\nwiglSBj3u/sjKQ5ZBuyasN4v3JYc52R3H+LuQ8p66LFaEZFsiVKbmGFmPzezXc2sPP5q6YXDJ6Pu\nBN519xsaOOwx4DQLDAPWubuapkREciRKTeOk8P3chG0OfLWF1x4O/BB4y8xeD7ddDvQHcPdJwOME\nj9t+QPDI7ektvKaIiLRAo0nD3XfPxoXd/QUamZfDgwGRzk13jIiItJ4odxqY2b7A3kDd/KHufm+2\nghIRkfzUaNIwsyuAEQRJ43GCDncvEPTmFpEC5A6JY3cmr4s0JEoh/ETgCGClu58OfAPolv4jIpKv\nXpyyLx+/3Iv4aPju8PHLvfhkbs/cBiYFIUrz1CZ3j5lZjZl1BVZR/zFYESkQ7rDLwNXs9s1VxGqh\nqBhitTBg6KeY6Y5DGhclacwxs52A24G5QBXwUlajEpGsMIPdh31KrBZiNcXEaoLtJe1r6/aLpBPl\n6alzwsVJZvZvoKu7v5ndsEQkW8zCO4ya7beLNCbdfBqD0+3TEOUihck9aJJKtT1dcVxNVwLp7zSu\nD987AEOANwj6VewHzAEOzG5oIpJp8aL3bt9cRVFJbV1Nw2NBQnCC99qaMEEUUVfr2Ly+He7Qaaet\nuf4xJIcafHrK3Ue6+0iCEWUHh2M77U8wGu124z+JSP4zg5WVPfnkta9QVLytqWrRq71YPL9nXYKI\n1RQze8q+zLq1AneYdWsF/7p6GFs2tKt76kp2TFEK4QPd/a34irsvMLP/yGJMIpJFB41bUK+pKV4c\nT1wvaV9LzaZS5j0ykHmPDARg8AmV7NSvSk1UO7go/TTeNLM7zGxE+LodUCFcpIAl/+FPtT7i3Pn1\nto04d74ShkRKGqcDbwM/C1/voIEDRfJKcpNRU9dTnW/WrRX1tj1zS0W9z6mZasfUaNJw983u/nt3\nPz58/d7dN7dGcCLSuHee2q2uSA3bitbvPLUbEPQAj9cm4vtn3VrBi1P2TXm++P55jwxk8AmVXDhz\nKuX91/HsnyrqEkdj55C2q9GkYWbDzWy6mb1nZh/FX60RnIik5w6fvlfOv64exhdLy3CHL5aW8a+r\nh/Hpe+XEYrClKqhNJBa15z0ykC1VpSnvFsygfVk1g0+orGui2m3ISgA+mdsbaPwc0naZN/IvbmYL\ngQsIeoPXPd3t7p9lN7Tm6V/R3y96+qJchyHSahITQVz8D378aah0+9OdN77fHZ65tYL5TTyHFI4J\n5RPmuvuQxo6LUtNY5+5PuPsqd/8s/spAjCKSAY0VrZtb1E7cbwYjVRgXoiWNZ8zsd2Z2oJkNjr8y\ncXEzu8vMVpnZggb2jzCzdWb2evj6ZSauK5LPmlO0nn5j/aL1U7+vX8N4+ub0RW2AWKzh9VSF8cQ6\niew4oiSNAwh6hF9N0Ev8euD/MnT9KcDRjRzzvLsPCl9XZei6InmpOUXri/v9N0/8+mD+89tB0bpr\n73U88euDubjffxOLwd3jjmH6DUPZedd1XDhzKhXHV/LsnyqYcvoxdde57yejufOUMXWJIhaDJ68b\nyoqFO9fVSUo6VnPGPdO4cOZUzrhnGp8t6aLEsQOK8vTUyBSvwzNxcXd/DlibiXOJFDr3pheta2og\nVluEuzHjhgOoroaFM78KGDVbi6muhtUf7kz1plJqthYH56hrUgoWYrHgaav3nutflzievG4oQ8cu\npENZMGRIx503MXzcgrrOfTv1q+LYy1+m155r1US1g2m0EA5gZscC+1B/uteMfOs3swHANHff7quU\nmY0AHgaWAsuBn7v72+nOp0K4FLLmFK2rq+EXXz2brRvb1W0rblfLV/b8jJKS4Jxdd6li/cqyunNU\nnFDJyIRzxmJw5yljeO+5/gD85MF/0H3AF+zUd2O9YrgGMGy7MlYIN7NJwEnATwm+mnwf2K3FEUYz\nD9jN3b8B3Az8vYEYx5vZHDObU7WmqpVCE8m85hStS0vhNx9Nqrftt4tvpaRk2znPun9avXOMTDpn\nURGcef+0uvWd+6+vlzDi50mOVXY8UWoaB7n7acDn7n4lwei2e2Y3rIC7r3f3qnD5caDUzHqkOG5y\nOKDikLIeZa0RmkgkmeiJnVw3SC5Yb9kS3Gkkuqz/udTUbDvn7WPH1DvHzJvrn7O2NrjTiPt8cVe+\nWNZZ9QrZTpSksSl8/9LM+gDVQO/shbSNme1iFnyfMbOhBPHqcV8pCJnoiT34hMp6NY7kgvWWLXBp\nv/PYurEd7Tpv5bqVf8CKY9RuLWbVe90574mpbFjViXdn7E7N1iIumDGVnXZdx79/eyD/N/Jk3IOE\ncd3Bp7Dw6d3Y89DFXLvkNt5/oS+x2mIlDtlOlKQxLZzu9XcEzUWLgAcycXEze4Bg6tiBZrbUzM40\ns7PNLP616URggZm9AfwBONmjFGFEcqw5Re3kntjxpqrBJ1TSvqy6bniQxIL1vePG4LWGFcf41QeT\nKC2FvUd/CDhFxTFKSsIJl9zYuLYDsRisqiynZmsJ6z/tTCwGz/2xgg2rOtPjq19wxp+nUVQER138\nKq8+sBebq9qpGUrqidIjvL27b4kvExTDN8e35RsVwiVfZKIndvJ6csEaYM9DF3PalGm0b7/t+Kd+\nX8GCaQPrPlO9qZjPl3ar+0z5ruso6VBLUdG2uA797/kUF2+7bixG3X5p+zLZI/yl+IK7b3H3dYnb\nRCS1TPTETl5PLlhDsB5PGPHjR18wv95nLnnx/nqfuXj2/fUSwohz6yeM+OdEkjX4n0VYT9gf6Ghm\nFQm9wUcAnVotQpE8kYmi9tO3bF+ATi5sxwvYqdZjMZh88ph6++/4wRhqE+b8Tu4BHovBdcNPqfeZ\n6w4+pd511UlPokr3XeIogp7f/djWE/x64ELg8uyHJpI/MlHU3rCmI09cfWBd4qithV989Sf84ms/\nqfsDfs+PRzNxn7O4b/xoAO4bH6zf8+PRwR//Q8aycObudOy2mWuX3MYehyzm3Rm7M3HfM6itDa6b\n2AN8wvSp1Gwu5tP3utOh62Z++8ltlPdfx6r3yqnZUsyE6dsX20XSSTdH+D3hHOHj3P3whN7gx7n7\nI60Yo0hOZaKoHYtBu05bqdlSzKzbKojF4NqDTmHzhvZsWt+O28eOoaYG3n+uP1WrO1H5bH+2boXK\nZ4P195/rT20tfLG0K7jRsesW3OFrw5dRXFrL5nUdePa2ILbEHuBmUNKhlpL2NQwYupLiYjhk/Bt8\nZc+19PvP1RQX1y+2q+gtjYlSCP8ZcDewAbgdGAxc6u5PZT+8plMhXLIhE0Xt2lq4dvgprH6/vG5/\nz6+vpfuAdXzwwrbCdvuyrWxa346ioqBpqWPXrWyp2tbbO7mIPej4SgyY/+jAumsm9wAf9N1KDv/p\n/HqxJNYw1LtbMlkIP8Pd1wOjge7AD4FrWhifSEHJRFG7uBgumV2/IH3Ji/fz4wfqF7Ynvn1HXUIo\nKgrWEyUXsQ8/bz4jz9sWW6oe4IkJIx5LQ3GKpBMlacT/c/oWcG849pP+E5MdSnOHBk8uel97UP2C\n9DUHnsLtY+sXtifufVa90WYn7n1Wvf3XDa9fxH76lmAa1sRr3nlK/R7gmt9bMiVK0phrZk8RJI0n\nzawLEGvkMyJtRpSe2qkkFs9ra2HiPmew6v3udO65kf9bdTM9vraW1R90593pu/P1gxfz209uo6g4\nRtWaTnTsspXffHwbHbtspWpNJ4qKY1y96DY6dN3Mp+91p2ZzUMQedHwl068fyvQbhlJxfCUXzJhK\nt12qeO+5/nTbpYoLZmh+b8msKEnjTOBS4Jvu/iXQDjg9q1GJ5JHGemqnatpJLp6bQfXmUjCn/6BV\nmMGwH76DFTklHWo56y/TKC6G7gO+oKg4Rrc+VZSWQrc+VRQVx+g+4AuKi2HA0JWUtK+pq2mYBa+d\n+m5gxLnzKSqCvY9axJ6HLmbv0Ysw0/zeklkNFsLNbBd3X5n2wxGOaW0qhEu2NHVo8OTiuTt07VXF\n+k8TCtTHVzLinG0d69xhxh8qePMf2wru+32nklHnb5vv++mbK3j979v2V5wQnCOxzpHYm1vze0sU\nmSiEPx7hOlGOEWkTmjo0eHLx3AzO+ktSgfq8+j2xzWDU+fUL7vGEEd9/+E/r7x957vztem8nrmt+\nb8mkdEnjG2a2Ps1rA9CrtQIVSaepvbVbIw53titQ35FUoE6uiTRWcG9OQV7ze0smpevcV+zuXdO8\nurh739YMViSVpvbWbo04gp7Z3+Kp64eyc//1XDBjKl13qaLy6QHUVgdDlCcX0xsruMdiTS/IN7eI\nL9KQklwHINISiQVnCJpdEv9ItlanteQ4DjtnPqs/7MbWL0up2Rx8NytpV0tpx2p6fu3zek1XicX0\nVAX3+PaiovT7U/2cDRXx031GJJ1Ic4QXEhXCdzzN7a2d7ThSFr6TemanSmqNFdybM1e35veWxmSy\nR7hIXmtub+3k0WWbug71axhmwR1G4npy4XvkedvH1dj3tsYK8FH++Gt+b8mUSEnDzIrNrI+Z9Y+/\nMnFxM7vLzFaZ2YIG9puZ/cHMPjCzN81scCauK21Lcwq9ydOmxic3uu8noyPth+1rKfFjqtZ0rIsr\nsWd21ZqO9c7pDlNO/xZTTj8m5/UYkagaTRpm9lPgU2A68K/wNS3th6KbAhydZv8xwB7hazzwxwxd\nV9qI5hR6Y7Htp02Nz4a3eX07amrS74/Ftu+8l3hMn31WM2F6/Z7ZE6ZPpc8+q+ud85lbK1g0ZxcW\nzeld11tbHe8k30UZ5fYD4AB3/ywrAZgNAKa5+3ZfrczsT8Asd38gXK8ERrj7iobOp5rGjufFKfuy\npaq0rkkq/se3fVk1B41LeRPb4LSpZ94/rW502XT7YfsaRtWajvTZZ3XdMbPv3pd3nhrA3kctYvi4\nBXXnXP52T8p6bAKCjnn4thFqQR3vJDei1jSiJI1ngCPdvSbtgc3USNKYBlzj7i+E6zOBS9x9TtJx\n4wnuRNi53877X/HmFdkIVfJYcwq9sRhcsus5devXLrltu17V6fbHr3PDESfXrV8wY2qDPbPj678f\nte34C2dOBeqf48KZU5UwpNW1uBBuZhea2YXAR8AsM7ssvi3cnjfcfbK7D3H3IWU9ynIdjuRAUwu9\n8W/9iVLVMBraD6lrKfGJkOISE4Z7sD/RM7fWH6EW1PFO8lu6mkaX8LWYoJ7RLmFba/1lXgbsmrDe\nL9wm0myJTU97HrqYa5fcxp6HLq6rN9TUpN8fr2k0pZaS6viKEyp5dlIFz/6pgorj1fFOCkODnfvc\n/UoAM/u+u/8tcZ+ZfT/bgYUeA84zs6nAAcC6dPUMkSiKiqBD1631ahRn3j+NO08ZQ4euWykpSb8/\nfvfQlE5zqTrZjTx3Pp/M6Q143aO46ngn+S5KTWOeuw9ubFuzLm72ADAC6EHwhNYVQCmAu08yMwNu\nIXjC6kvg9OR6RjIVwiWqVPWGpqxD80a+TT4e1PFOci9qTaPBOw0zO4Zg4qW+ZvaHhF1dgYwUxd19\nbCP7HTg3E9cSSZZuZNgo69C8kW8bO14JQ/JZurGnlgNzgePC97gNwAXZDEpERPJTuprGG8AbZna/\nu1e3YkwiIpKn0jVPvQV4uLzdfnffL3thiYhIPkrXPBV/SD1eU7gvfD+VMJnko6+8voQJ5T/jxrU3\n5ToUEZE2J13z1CcAZnakuyf2PrrEzOYBl2Y7uOZyDMrBcCUPEZEMijIJk5nZcHefHa4cRIEMqe4Y\nVh7cFCl5iIi0XJSkcSZwl5l1Awz4HDgjq1FlkBPWY3TnISLSYo0mDXefC3wjTBq4+7qsR5UlarYS\nEWmZdE9Pneruf04enDD+JJW735Dl2LJGzVYiIs2T7k6jc/jepTUCaW1qthIRabp0T0/9KVy81t03\nt1I8OaFmKxGRaKIUwheY2afA8+HrhUKua6TTVpLHyYdfT8c1Vdtt39SjjKkazFFEWiBKIfzrZtYf\nOAQ4FrjVzL5w90FZjy5H4snjprU35jqUZum4popNKSajSpVIRESaotH+FmbWDxhOkDQqgLeBB7Mc\nV174WfmE7daTt4mI7EiiNE8tBl4Drnb3s7McT95JlSR+Vj6hYO9CRERaIkrP7grgXuAHZvaSmd1r\nZmdmOa68N6H8Z7kOQUSk1UWpabxhZh8CHxI0UZ0KHAbc2dKLm9nRwE1AMXCHu1+TtH8c8Du2zQt+\ni7vf0dLrZoJj3ETh3G2UV66keGstp+93Zb3tKo6LSFM0mjTMbA7QHniR4OmpQ+ODGbaEmRUDtwJH\nAkuB18zsMXd/J+nQB939vJZeLxvytZlqU4+y7YrexVtrqW1XvF2BXMVxEWmKKDWNY9x9dRauPRT4\nwN0/AjCzqcB3gOSkkdcSax75kkBS3Tmcvt+VKZ+oEhFpiijNU9lIGAB9gSUJ60uBA1Ic9z0zOxR4\nD7jA3ZckH2Bm44HxAP2zEGhU+ZhAREQyKd+HOP8nMCCcJXA6cE+qg9x9srsPcfchPVs1vIbp0VwR\naYuiNE+q3Yb9AAAP/0lEQVRlyzJg14T1fmwreAPg7p8lrN4BXNcKceWdxnp4/7T7BZinnkxxQ7+d\nAeiy9HO6LPucWElxs64hIgLpR7k9Id0H3f2RFl77NWAPM9udIFmcDPwgKYbe7r4iXD0OeLeF12xV\nmZp2trEe3uaOJ8zjnphA4p/rsuzzlJP0xjepF7mIRJHuTuPbafY50KKk4e41ZnYe8CTBI7d3ufvb\nZnYVMMfdHwPON7PjgBpgLTCuJddsbfk0flX8DmPNPn3qbVdSEJGmSDfK7enZvri7Pw48nrTtlwnL\nlwGXZTsOERGJJlJNw8yOBfYBOsS3uftV2QpKRETyU5TOfZOATsBIgmL0icCrWY5LEnRZ9jldln6e\nct/54VNaDRXC44pqasGhx9vL621P/ykRkfqi3Gkc5O77mdmb7n6lmV0PPJHtwApdRvtpOGBJ6/FF\ns7QJo65mkXyOUHxTql7k8e0iInFRksam8P1LM+sDfAb0zl5IksyLi6htt+1R2ZJN1Q0eu2rQtqeY\nO66p4u43rwAa7hEeTxR6rFZEoojSuW+ame1EMHDgPGAR8EA2g2oLNAquiLRFUZLGde7+hbs/DOwG\n7AX8OrthFT5P1RYkIlLgojRPvQQMBnD3LcAWM5sX3yZN01jP63P6/g/FW2vr7bPaGCWbYinPl1zP\n+Mrr9YfmOj9hOJMuSz8nVqoe4SLSfOl6hO9CMKhgRzOrYFvNtCvB01TSDI31vI4PYZ6ooYSRCeoR\nLiJNke5O4yiCHtj9gBsStq8HLs9iTG1GPs63oR7hItIS6XqE3wPcY2bfC+sZIiKyg4tSCJ9tZnea\n2RMAZra35giPTkOki0hbEiVp3E0wqGC8XeM9QH8JM6S8ciU93l5O2Yp1nL7flUHRe3M1xZtrch2a\niMh2oiSNHu7+VyAGwei0QG36j0iixLuNeM/r+Cv+pFR8/m43Aw+eiireWrvdk1QtlXjtxOJ3clzJ\n+0VEINojtxvNrDvhgzZmNgxYl9Wo2rDkx1eTe2qv/kY/oH5v7mTJn0l+zDbRHyIW4vVYrYhEESVp\nXAg8BnzNzGYDPQkGLRQRkR1Mo81T7j4POAw4CPgJsI+7v5mJi5vZ0WZWaWYfmNmlKfa3N7MHw/2v\nmNmATFw3FzSsiIi0BY0mDTPrAJwP/Aq4Ejg33NYiZlYM3AocA+wNjDWzvZMOOxP43N2/DvweuLal\n180VDSsiIm1BlEL4vQQTMN0M3BIu35eBaw8FPnD3j9x9KzAV+E7SMd8B7gmXHwKOMLOC/eub6m6j\nOQXo5M+IiLSWKDWNfd098Q7gGTN7JwPX7gskVnCXAgc0dEw4p/g6oDuwJgPXzwvNKUCraC0iuRLl\nTmNe+MQUAGZ2ADAneyE1nZmNN7M5ZjZnda6DERFpw6Ikjf2BF81skZktIhj19ptm9paZtaQgvgzY\nNWG9X7gt5TFmVgJ0I5gEqh53n+zuQ9x9SM8WBJRtqmuISKGL0jx1dJau/Rqwh5ntTpAcTgZ+kHTM\nY8CPCBLVicDT7o1Mhi0iIlnTaNJw90+yceGwRnEewRAlxcBd7v62mV0FzHH3x4A7gfvM7ANgLUFi\nKWj5OPKtiEhUUe40ssbdHwceT9r2y4TlzcD3WzsuERFJLUpNQ0REBFDSyAkNly4ihUpJQ0REIlPS\nEBGRyJQ0ckRNVCJSiJQ0REQkMiUNERGJTElDREQiU9LIIdU1RKTQKGmIiEhkSho5pmlgRaSQKGnk\nmIZLF5FCoqSRB3S3ISKFQkkjD+huQ0QKhZKGiIhEpqSRJ/T4rYgUgpwkDTMrN7PpZvZ++L5zA8fV\nmtnr4eux1o5TRETqy9WdxqXATHffA5gZrqeyyd0Hha/jWi+83NDdhojku1wlje8A94TL9wDfzVEc\neUeJQ0TyWa6SRi93XxEurwR6NXBcBzObY2Yvm5kSi4hIjpVk68RmNgPYJcWuXySuuLubmTdwmt3c\nfZmZfRV42szecvcPU1xrPDAeoH8L484HPyufwE1rb8x1GCIi28nanYa7j3L3fVO8/gF8ama9AcL3\nVQ2cY1n4/hEwC6ho4LjJ7j7E3Yf0zMpP0/rUTCUi+ShXzVOPAT8Kl38E/CP5ADPb2czah8s9gOHA\nO60WYR5QT3ERyTe5ShrXAEea2fvAqHAdMxtiZneEx/wHMMfM3gCeAa5x9x0qaainuIjkm6zVNNJx\n98+AI1JsnwOcFS6/CPxnK4cmIiJpqEd4nlNtQ0TyiZJGAVDiEJF8oaRRIJQ4RCQfKGmIiEhkShoF\nRHcbIpJrShoFRolDRHJJSaMAKXGISK4oaRQoJQ4RyQUljQKmxCEirU1Jo8ApcYhIa1LSaAOUOESk\ntShptBFKHCLSGpQ02hANpS4i2aak0YZoKHURyTYljTZGzVQikk3m3tD03IXJzDYAlbmOI6IewJpc\nBxFBocQJhRNrocQJhRNrocQJ+Rnrbu7e6IzZOZmEKcsq3X1IroOIwszmFEKshRInFE6shRInFE6s\nhRInFFasydQ8JSIikSlpiIhIZG0xaUzOdQBNUCixFkqcUDixFkqcUDixFkqcUFix1tPmCuEiIpI9\nbfFOQ0REsqRNJQ0zO9rMKs3sAzO7NNfxNMTM7jKzVWa2INexpGNmu5rZM2b2jpm9bWZ52+XczDqY\n2atm9kYY65W5jikdMys2s/lmNi3XsaRjZovM7C0ze93M5uQ6noaY2U5m9pCZLTSzd83swFzHlMzM\nBoa/x/hrvZkVXMeqNtM8ZWbFwHvAkcBS4DVgrLu/k9PAUjCzQ4Eq4F533zfX8TTEzHoDvd19npl1\nAeYC383T36kBnd29ysxKgReAn7n7yzkOLSUzuxAYAnR19zG5jqchZrYIGOLu+danoB4zuwd43t3v\nMLN2QCd3/yLXcTUk/Hu1DDjA3T/JdTxN0ZbuNIYCH7j7R+6+FZgKfCfHMaXk7s8Ba3MdR2PcfYW7\nzwuXNwDvAn1zG1VqHqgKV0vDV15+IzKzfsCxwB25jqUtMLNuwKHAnQDuvjWfE0boCODDQksY0LaS\nRl9gScL6UvL0D1whMrMBQAXwSm4jaVjY5PM6sAqY7u75GuuNwMVALNeBRODAU2Y218zG5zqYBuwO\nrAbuDpv87jCzzrkOqhEnAw/kOojmaEtJQ7LEzMqAh4EJ7r4+1/E0xN1r3X0Q0A8YamZ51/RnZmOA\nVe4+N9exRHSwuw8GjgHODZtW800JMBj4o7tXABuBfK5ptgOOA/6W61iaoy0ljWXArgnr/cJt0gJh\nfeBh4H53fyTX8UQRNk08Axyd61hSGA4cF9YKpgKHm9mfcxtSw9x9Wfi+CniUoBk43ywFlibcWT5E\nkETy1THAPHf/NNeBNEdbShqvAXuY2e5hJj8ZeCzHMRW0sLh8J/Cuu9+Q63jSMbOeZrZTuNyR4IGI\nhbmNanvufpm793P3AQT/jT7t7qfmOKyUzKxz+AAEYXPPaCDvnvhz95XAEjMbGG46Asi7hzUSjKVA\nm6agDQ1Y6O41ZnYe8CRQDNzl7m/nOKyUzOwBYATQw8yWAle4+525jSql4cAPgbfCWgHA5e7+eA5j\nakhv4J7wqZQi4K/untePsxaAXsCjwXcHSoC/uPu/cxtSg34K3B9+YfwIOD3H8aQUJt8jgZ/kOpbm\najOP3IqISPa1peYpERHJMiUNERGJTElDREQiU9IQEZHIlDRERCQyJQ3JO2Y2zsz6RDhuipmdGHV7\nBuK6PGF5QJRRisNYPjazs9McM8jMvpXBOMeZ2S0tPMcsMxsSLj8e7wPTgvONiI/oa2YnhSNR65Ho\nAqSkIfloHNBo0siByxs/JKX/cfdJafYPAjKWNJrKzNL213L3b2VyAEB3fxA4K1Pnk9alpCFZFX4j\nX2hm94fzHDxkZp3Cffub2bPhYHhPmlnv8A5hCEFHrdfNrKOZ/dLMXjOzBWY2OeypHvX6210j3D7L\nzK4N5+B4z8wOCbd3MrO/WjCHyKNm9oqZDTGza4COYUz3h6cvNrPbLZi/46mwJ3pj8Xw//DneMLPn\nws5oVwEnhec+ycyGmtlL4eB7L8Z7Ood3EI+Y2b/N7H0zuy7hvKeHP8erBJ0y49u/Hf4M881shpn1\nCrdPNLP7zGw2cF/4e54a/hs9CnRMOMciM+thZmfbtrkgPjazZ8L9o8N455nZ3ywYqyw+v81CM5sH\nnBD130zynLvrpVfWXsAAgpFSh4frdwE/Jxi6/EWgZ7j9JIJe/ACzCOZwiJ+jPGH5PuDb4fIU4MQU\n15wCnBjhGteHy98CZoTLPwf+FC7vC9TEYwGqkn6uGmBQuP5X4NSGYklYfwvoGy7vFL6PA25JOKYr\nUBIujwIeTjjuI6Ab0AH4hGC8td7AYqAn0A6YHT8fsDPbOvGelfAzTySYH6VjuH5hwu9mv6SfexHQ\nIyG+UuB54NtAD+A5grlMAC4BfhnGtwTYA7Dw9zMt4RwjEtf1KpxXmxlGRPLaEnefHS7/GTgf+DfB\nH+Xp4Y1DMbCigc+PNLOLgU5AOfA28M8I1x3YyDXiAzDOJUgCAAcDNwG4+wIzezPN+T929/jwKonn\nSGc2MMXM/ppw/WTdCIZE2YMg4ZYm7Jvp7usAzOwdYDeCP9yz3H11uP1BYM/w+H7Ag+EdVjvg44Rz\nPebum8LlQ4E/ALj7m4383DcRjJn1TwtG7d0bmB3+jtsBLwF7Efx+3g9j+jOQr0OrSxMoaUhrSB6r\nxgm+fb7t7mmn5TSzDsBtBN96l5jZRIJvsVE0do0t4Xstzft/YUvCci0JTToNcfezzewAgkmY5prZ\n/ikO+xXwjLsfb8E8JrPSXLOxuG8GbnD3x8xsBMEdRtzGxuJNZmbjCBLVefFNBHOXjE06blBTzy2F\nQTUNaQ39bduczT8gmIq1EugZ325mpWa2T3jMBqBLuBxPEGvCtvKmPBWV7hoNmQ38V3j83sB/Juyr\ntmCo+GYzs6+5+yvu/kuCiYN2pf7PC8GdRnxY/3ERTvsKcJiZdQ/j+34D5/pRmnM8R/BvgwXzkOyX\nIvb9CZrvTnX3+ARSLwPDzezr4TGdzWxPghGGB5jZ18LjxiafTwqTkoa0hkqCCXzeJWhj/6MHU/Ke\nCFxrZm8ArwMHhcdPASZZMLLuFuB2giG5nyQYAj+SRq7RkNsIEs07wK8JmsLWhfsmA28mFMKb43dm\n9pYFj+u+CLxBMPfH3vFCOHAd8Fszm0+EOyB3X0FwB/ESQdJ7N2H3ROBvZjYXSDfP9x+BsvDf6CqC\n5rZk5xE0Dz4TxnpH2CQ2DnggbNJ6CdjL3TcTNEf9KyyEr2rs55DCoFFuJavC5pVp7p53s+ilYsHQ\n6qXuvjn8ljwDGBgmoOacbwrBz/9QBsMseGFT2c/dfUyuY5GmUU1DpL5OBN+kSwna689pbsIIrQN+\nZWY9PH1fjR1GeDd1BanvZiTP6U5DREQiU01DREQiU9IQEZHIlDRERCQyJQ0REYlMSUNERCJT0hAR\nkcj+P/1Va4xL9B37AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1152c4c10>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_decision_regions(X_combined, y_combined, classifier=knn, test_idx=range(105, 150))\n",
    "plt.xlabel('petal length [standardized]')\n",
    "plt.ylabel('petal width [standardized]')\n",
    "plt.legend(loc='upper left')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "\n",
    "我们设置k=5，得到了相对平滑的决策界。\n",
    "\n",
    "k的选择对于KNN模型来说至关重要，除此之外，距离度量也是很有用的。通常，欧氏距离用于实数域的数据集，此时一定要对特征进行标准化，这样每一维度特征的重要性等同。我们在上面的代码中使用的距离度量是'minkowski',它是欧氏距离和曼哈顿距离的一般化：\n",
    "\n",
    "\n",
    "![](https://ooo.0o0.ooo/2016/06/22/576a40dc59014.png)\n",
    "\n",
    "\n",
    "\n",
    "如果p=2，则退化为欧氏距离，p=1，则退化为曼哈顿距离。使用metric参数可以选择不同的距离度量。\n",
    "\n",
    "\n",
    "**Note** 维度诅咒\n",
    "\n",
    "注意，如果特征维度过大，KNN算法很容易过拟合。我们可以想象，对于一个固定大小的训练集，如果特征空间维度非常高，空间中最相似的两个点也可能距离很远(差别很大)。虽然我们在逻辑回归中讨论了正则项来防止过拟合，但是正则项却不适用于KNN和决策树模型，我们只能通过特征选择和降维手段来避免维度诅咒。下一章我们会讲到。\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## 总结\n",
    "\n",
    "本章，你学习了许多不同的机器学习算法，用于解决线性和非线性问题。如果我们关注模型可解释性，决策树是很好的选择。逻辑斯蒂回归不但可以在在线学习场景下大展拳脚还能预测概率。虽然SVM能解决线性和非线性问题，但是它参数个数比较多，调参挺麻烦。集成算法包括随机森林则不需要调节过多的参数，也不会像决策树一样容易过拟合，这使得它很受欢迎。KNN通过懒惰学习进行分类，他不需要模型训练的过程但是在预测时的计算成本相对比较高。\n",
    "\n",
    "\n",
    "然而，比选择合适的算法更重要的是训练集数据本身。如果数据的特征不够好，再好的算法也没用。\n",
    "\n",
    "\n",
    "在下一章，我们会讨论预处理涉及到的内容。\n",
    "\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
